This blob post goes through the different methods of connecting an HPC cluster in a box (within docker container) so that the network performance is worth the effort and all containers are addressable. It's going to talk about VXLAN, MACLAN and some pipework to glue them together.
Happy new year, y'all!
To kick of 2016 I am going to contribute to FOSDEM by talking about a Multi-host containerised HPC cluster. And since the HPC Advisory Council was so nice to provide an actual HPC resource I will present a 2.0 version of the talk (lessons learned after FOSDEM) at the HPC Advisory Council Workshop in Stanford.
In this post we will spin up a kvstore holding Consul and connect two distinct docker-machines to the Consul cluster to share the networking configuration.
As I wrote last month the zeromq plugin in 1.4 has an issue. And it's still not usable since this patch is not applied. :(
The zeromq filter within logstash 1.4 is not working out as expected for me. I opened an issue on github to cope with that. For now I work around this issue by starting logstash 1.3 as a separate instance and let this version deal with zeromq.
For those of you asking WTF?... :) ZeroMQ is a message library that provides multiple patterns like PUB/SUB, PUSH/PULL and others. I got a use-case in which I want specific log events to be handled outside of logstash. And more outside then firing up the ruby filter. I want to process the event within a external daemon to check some things, update the JSON with additional information (lookup names, routes or alike) and after I am done I push it back into the logstash pipeline.