logstash

The second M.E.L.I.G. MeetUp was about the famous ELK stack (Elasticsearch / Logstash / Kibana).

Again, we were kindly hosted by InnoQ (they have a podcast). Thanks for that...

OK guys, since the ELK is quite popular on docker-hub, I was wondering if I could push it one notch further. Turns out I could, please welcome qnib/monster...

git (master) $ git clone https://github.com/ChristianKniep/docker-monster.git
git (master) $ cd docker-monster
docker-monster (master) $ docker-compose up -d
Creating dockermonster_monster_1...
docker-monster (master) $

As I wrote last month the zeromq plugin in 1.4 has an issue. And it's still not usable since this patch is not applied. :(

Since I was ask on hub.docker.com if my qnib/elk image is going to provide kibana4 in the near future I figured it would be worth to blog about it.

The image in question is quite nice for trying the ELK stack out and I take some pride in stating that it's the number 2 image popping up if you search for 'elasticsearch' (and rank by stars). :)

The zeromq filter within logstash 1.4 is not working out as expected for me. I opened an issue on github to cope with that. For now I work around this issue by starting logstash 1.3 as a separate instance and let this version deal with zeromq.

Come again?!

For those of you asking WTF?... :) ZeroMQ is a message library that provides multiple patterns like PUB/SUB, PUSH/PULL and others. I got a use-case in which I want specific log events to be handled outside of logstash. And more outside then firing up the ruby filter. I want to process the event within a external daemon to check some things, update the JSON with additional information (lookup names, routes or alike) and after I am done I push it back into the logstash pipeline.

If you are looking for an excuse to use logstash your local webserver is low hanging fruit.

Someone accesses your website and your web server will store some details about the visit:

10.10.0.1 - - [29/Oct/2014:18:42:18 +0100] "GET / HTTP/1.1" 200 2740 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4"
10.10.0.1 - - [29/Oct/2014:18:42:19 +0100] "GET /css/main.css HTTP/1.1" 200 2805 "http://qnib.org/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4"
10.10.0.1 - - [29/Oct/2014:18:42:19 +0100] "GET /pics/second_strike_trans.png HTTP/1.1" 200 29636 "http://qnib.org/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4"