Running Docker services on single nodes is quite boring, so let's boot up three boxes with the latest version of docker.

This is a blog post to accompany the talk I gave a the Berlin Docker Meetup:

Create multi-host Docker SWARM

For this little the boxes are only equipped with 1GB of RAM and one core, for Elasticsearch this is quite small... If you like to change it look into the Vagrantfile.

$ git clone https://github.com/qnib/vagrant-orchestration.git
Cloning into 'vagrant-orchestration'...
remote: Counting objects: 11, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 11 (delta 1), reused 7 (delta 0), pack-reused 0
Unpacking objects: 100% (11/11), done.
Checking connectivity... done.
$ cd vagrant-orchestration/docker-1.12
$ vagrant up
Bringing machine 'swarm0' up with 'virtualbox' provider...
Bringing machine 'swarm1' up with 'virtualbox' provider...
Bringing machine 'swarm2' up with 'virtualbox' provider...
==> swarm0: Importing base box 'btexpress/ubuntu64-16.04'...
==> swarm0: Matching MAC address for NAT networking...
*snip*
==> swarm2:   inflating: consul-template
==> swarm2: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /lib/systemd/system/docker.service.
==> swarm2: export DOCKER_HOST=:2376 ; unset DOCKER_TLS_VERIFY ; unset DOCKER_CERT_PATH
$

Now we just need to create a swarm out of them:

$ export DOCKER_HOST=192.168.100.10:2376
$ docker swarm init --advertise-addr=192.168.100.10
Swarm initialized: current node (4qoc5xmwadimvvlkpw49vyau2) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-3lzna54jxjosmfbxprg3shcm9qspp6js50xgr9d9mnz6t67t26-b6chdyqxm7xfa0oun4jyu5a1u \
    192.168.100.10:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
$ docker -H 192.168.100.11:2376 swarm join --advertise-addr=192.168.100.11 --token SWMTKN-1-3lzna54jxjosmfbxprg3shcm9qspp6js50xgr9d9mnz6t67t26-b6chdyqxm7xfa0oun4jyu5a1u 192.168.100.10:2377
This node joined a swarm as a worker.
$ docker -H 192.168.100.12:2376 swarm join --advertise-addr=192.168.100.12 --token SWMTKN-1-3lzna54jxjosmfbxprg3shcm9qspp6js50xgr9d9mnz6t67t26-b6chdyqxm7xfa0oun4jyu5a1u 192.168.100.10:2377
This node joined a swarm as a worker.
$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
07qihqioh09i92apn9zurnpu0    swarm2    Ready   Active
4qoc5xmwadimvvlkpw49vyau2 *  swarm0    Ready   Active        Leader
9wyqn6x9c9nkmuawxmll9d60p    swarm1    Ready   Active
$

Docker Services

The goal in this post is to create docker services for...

  • Consul: To base stacks on
  • Elasticsearch: As a log backend
  • Kopf: As a frontend to check what is going on in ES

Consul

As the embedded DNS server only resolves complete service names and not the individual tasks in a deterministic manner (the tasks include the task-ID, which is not known in advance), I came up with a two step approach (see: Consul as a Service).

$ docker network create -d overlay consul-net
2133n6h6ke23c7k2li9frac0r
$ docker service create --name consul-blue --replicas=1 --publish=8501:8500 \
                            -e CONSUL_BOOTSTRAP_EXPECT=3 \
                            -e CONSUL_SKIP_CURL=true \
                            -e CONSUL_CLUSTER_IPS=consul-blue,consul-green \
                            --network consul-net \
                            qnib/alpn-consul@sha256:9006ef36441f8f45e01682a4f61ef213ab345dfde9680bb7e225c435e4c878e5
d24opk96vmlbxp5pueyxxmebq
$ sleep 15 ; docker service ls
ID            NAME         REPLICAS  IMAGE                                                                                     COMMAND
d24opk96vmlb  consul-blue  1/1       qnib/alpn-consul@sha256:9006ef36441f8f45e01682a4f61ef213ab345dfde9680bb7e225c435e4c878e5
$

After a couple of seconds an empty consul UI should be available at http://192.168.100.10:8501/ui/#/dc1/nodes.

After the seed is planted one group can be fully started.

$ docker service create --name consul-green --mode=global --publish=8500:8500 \
                            -e CONSUL_BOOTSTRAP_EXPECT=3 \
                            -e CONSUL_SKIP_CURL=true \
                            -e CONSUL_CLUSTER_IPS=consul-blue,consul-green \
                            --network consul-net \
                            qnib/alpn-consul@sha256:9006ef36441f8f45e01682a4f61ef213ab345dfde9680bb7e225c435e4c878e5
5k8fotte9ofs73v0rwky744vn
$

As the cluster reaches the quorum, it bootstraps itself.

Now we recreate the consul-blue service as well as a global service.

$ docker service rm consul-blue
$ sleep 15 # to wait for the tasks to finish
$ docker service create --name consul-blue --mode=global --publish=8501:8500 \
                            -e CONSUL_BOOTSTRAP_EXPECT=3 \
                            -e CONSUL_SKIP_CURL=true \
                            -e CONSUL_CLUSTER_IPS=consul-blue,consul-green \
                            --network consul-net \
                            qnib/alpn-consul@sha256:9006ef36441f8f45e01682a4f61ef213ab345dfde9680bb7e225c435e4c878e5
3xglxrb69iqmlhzfle00g4rcu
$

Et voila, we got six consul nodes...

$ docker exec $(docker ps --format '' |head -n1) consul members
Node          Address        Status  Type    Build  Protocol  DC
0415081a1d18  10.0.0.8:8301  alive   server  0.6.4  2         dc1
3461f4621444  10.0.0.9:8301  alive   server  0.6.4  2         dc1
34f41ff6b55f  10.0.0.6:8301  alive   server  0.6.4  2         dc1
88d0a6e5b04b  10.0.0.7:8301  alive   server  0.6.4  2         dc1
9099affaf49e  10.0.0.5:8301  alive   server  0.6.4  2         dc1
f203f17e4b59  10.0.0.3:8301  alive   server  0.6.4  2         dc1
$

Elasticsearch

Elasticsearch docks onto the consul services and runs in global mode.

$ docker service create --name elasticsearch --mode=global --publish=9200:9200 --publish=9300:9300 \
                            -e CONSUL_SKIP_CURL=true \
                            -e CONSUL_CLUSTER_IPS=consul-blue,consul-green \
                            --network consul-net \
                            qnib/alpn-elasticsearch@sha256:f06e9a6bacea23096306fc99df0d3edc11275c12b0fc85b24b835730345f9064
6x4kyxuisgq5jyyo4x3phyncm
$ docker service ls
ID            NAME           REPLICAS  IMAGE                                                                                            COMMAND
3xglxrb69iqm  consul-blue    global    qnib/alpn-consul@sha256:9006ef36441f8f45e01682a4f61ef213ab345dfde9680bb7e225c435e4c878e5
5k8fotte9ofs  consul-green   global    qnib/alpn-consul@sha256:9006ef36441f8f45e01682a4f61ef213ab345dfde9680bb7e225c435e4c878e5
6x4kyxuisgq5  elasticsearch  global    qnib/alpn-elasticsearch@sha256:f06e9a6bacea23096306fc99df0d3edc11275c12b0fc85b24b835730345f9064
$ sleep 45
$ echo -e 'NAME\t\t\t\t\t\tSTATUS' ;  docker ps --format '\t\t'
NAME                                        STATUS
elasticsearch.0.6urtk1lhkzmysnkf2i5sqp3o4   Up 4 minutes (healthy)
consul-blue.0.b0rd0ktvofswbe7lrkrsboo8g     Up 9 minutes (healthy)
consul-green.0.60v4dtutijhapp9pmz566g9zz    Up 12 minutes (healthy)
$

After some time the cluster should be p'n'running...

$ curl -s 192.168.100.10:9200/_cluster/health |jq .
{
  "cluster_name": "qnib",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 0,
  "active_shards": 0,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}
$

Kopf

To get an idea what is going on within the cluster, let us start a kopf service. Load balanced across two tasks should be enough.

$ docker service create --name kopf --replicas=2 --publish=80:80 \
                            -e CONSUL_SKIP_CURL=true \
                            -e CONSUL_CLUSTER_IPS=consul-blue,consul-green \
                            --network consul-net \
                            qnib/kopf@sha256:1536e4cfe052b8ff2268823996000bbe45edd5cdbb021997e22593391044ec36
4tgwrs3z9m418l1200xxig6g9
$ docker service ls
$ docker service ls
ID            NAME           REPLICAS  IMAGE                                                                                            COMMAND
3xglxrb69iqm  consul-blue    global    qnib/alpn-consul@sha256:900*snip*
4irbxjiid8rb  elasticsearch  global    qnib/alpn-elasticsearch@sha256:f06*snip*
5k8fotte9ofs  consul-green   global    qnib/alpn-consul@sha256:900*snip*
drv2gl2kq231  kopf           2/2       qnib/kopf@sha256:153*snip*
$$

Use It

Now let's push some data...

$ curl -XPUT 'http://192.168.100.10:9200/twitter/tweet/1' -d '{
    "user" : "kimchy",
    "post_date" : "2016-08-16T14:12:12",
    "message" : "trying out Elasticsearch"
}'
{"_index":"twitter","_type":"tweet","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}$

And kopf reports something back... :)

Future Work

From here on the sky is the limit... Put logstash next to it, or dump wikipedia into ES - kibana4 would be nice as well.

It's all so fresh. :)