Docker 1.13 Prometheus end-point and qcollect

Docker 1.13 is on it's way and I like what comes to light.

The highlights from where I stand are:

service port publishing now as mode host or ingress, which allows for service ports to be outside of the IPVS load-balancer and just exposed on the SWARM node.
the load-balancer seems to honour established connections
experimental has an end-point /metrics, which exposes Prometheus formatted metrics.

And this last bit got me interested. So much, though, that I hacked a Prometheus collector into qcollect. :)

Docker 1.13 experimental Vagrant Stack

I created a docker-1.13 vagrant stack with three small VMs.

$ git clone https://github.com/qnib/vagrant-orchestration.git 
$ cd vagrant-orchestration
$ vagrant up swarm0                                                                                                                                                                                         git:(master|✚1
Bringing machine 'swarm0' up with 'virtualbox' provider...
==> swarm0: Importing base box 'bento/ubuntu-16.04'...
==> swarm0: Matching MAC address for NAT networking...
*snip*
==> swarm0: Created symlink from /etc/systemd/system/multi-user.target.wants/qcollect.service to /lib/systemd/system/qcollect.service.
$

After that a SWARM cluster with only one node as formed.

$ export DOCKER_HOST=192.168.100.10:2376
$ docker node ls 
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
y54euwa9prtmvdfilxkb7fkds *  swarm0    Ready   Active        Leader
$

Not only instantiates the Vagrant provisioning the SWARM cluster, but also a metrics stack...

$ docker service ls 
ID            NAME      REPLICAS  IMAGE                                                                                       COMMAND
b6kbrioq3paq  influxdb  1/1       qnib/influxdb@sha256:8f560be674b3601194de4c80c126e90a55916983bc076bf9ae5c944378d123b5
dw31bqwdk9hu  http      global    qnib/httpcheck@sha256:59a2d0f3d2b31e83056f9290e05708ce5a78434a894953ed9f0b141b39c222b1
ig3b76a2ukp0  grafana3  1/1       qnib/alpn-grafana3@sha256:60e8c2cbe6edd438c210a89e7354f2cabd5cda0f94aba9f1ec25765772e1e7dc
u3371h61eu1b  consul    1/1       qnib/alpn-consul@sha256:e9a518ec4fd39fbcaf4f47192f20c03728f7050cb3f69cc40c1c57934a9ea6e8
$

... and qcollect-as-a-service...

$ vagrant ssh swarm0 -- sudo systemctl status qcollect                                                                                                                                                     git:(master|✚1…
* qcollect.service - QCollect metrics collector
   Loaded: loaded (/lib/systemd/system/qcollect.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-11-20 16:34:28 UTC; 20min ago
 Main PID: 6124 (qcollect)
    Tasks: 7
   Memory: 8.0M
      CPU: 3.991s
   CGroup: /system.slice/qcollect.service
           `-6124 /usr/local/bin/qcollect

Nov 20 16:54:25 swarm0 qcollect[6124]: time="2016-11-20T16:54:25.623326252Z" level=info msg="POST of 100 metrics to InfluxDB took 0.005613 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:28 swarm0 qcollect[6124]: time="2016-11-20T16:54:28.625531659Z" level=info msg="POST of 26 metrics to InfluxDB took 0.012559 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:28 swarm0 qcollect[6124]: time="2016-11-20T16:54:28.62863199Z" level=info msg="POST of 60 metrics to InfluxDB took 0.012831 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:30 swarm0 qcollect[6124]: time="2016-11-20T16:54:30.629644927Z" level=info msg="POST of 100 metrics to InfluxDB took 0.009841 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:33 swarm0 qcollect[6124]: time="2016-11-20T16:54:33.626255199Z" level=info msg="POST of 26 metrics to InfluxDB took 0.009733 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:33 swarm0 qcollect[6124]: time="2016-11-20T16:54:33.62657149Z" level=info msg="POST of 60 metrics to InfluxDB took 0.009258 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:35 swarm0 qcollect[6124]: time="2016-11-20T16:54:35.625379582Z" level=info msg="POST of 100 metrics to InfluxDB took 0.008441 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:38 swarm0 qcollect[6124]: time="2016-11-20T16:54:38.623678086Z" level=info msg="POST of 26 metrics to InfluxDB took 0.010786 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:38 swarm0 qcollect[6124]: time="2016-11-20T16:54:38.627324732Z" level=info msg="POST of 60 metrics to InfluxDB took 0.012158 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:40 swarm0 qcollect[6124]: time="2016-11-20T16:54:40.635768324Z" level=info msg="POST of 100 metrics to InfluxDB took 0.007869 seconds" app=qcollect handler=InfluxDB pkg=handler

Grafana3 can now be reached under http://192.168.100.10:3000/dashboard/db/docker-engine.

After starting the second and third node...

vagrant up swarm1 swarm2                                                                                                                                                                                 git:(master|✚1…
Bringing machine 'swarm1' up with 'virtualbox' provider...
Bringing machine 'swarm2' up with 'virtualbox' provider...
==> swarm1: Importing base box 'bento/ubuntu-16.04'...
*snip*
==> swarm2: This node joined a swarm as a manager.
==> swarm2: Created symlink from /etc/systemd/system/multi-user.target.wants/qcollect.service to /lib/systemd/system/qcollect.service.

... the dashboard shows stats from all three engines and the global service http, spans across the complete cluster.

That is all as an initial step...