Send Nagios events to Logstash

Logstash is an awesome swiss army knife to manage events and logs. I wanted to collect some Nagios events and send them into Logstash with zero programming as requirement (I want to avoid specific development by using reusable open source tools).

With our events managed by Logstash, we can imagine a lot of possibilities like adding some event management rules (refinement, filtering…), archiving in S3, send to IRC, extract and send metrics to graphite… In this tutorial we will use Logstash to print collected events from Nagios.

Basic Nagios installation

We will use a basic Nagios instance in a Debian 7 virtual machine. Vagrant is great tool to manage development virtual machines so we will use it with the lastest Debian that we can find on vagrantcloud.com.

I like the puphpet/debian75-x64 image but any debian based image is ok.

vagrant init "puphpet/debian75-x64"
vagrant up
vagrant ssh

On the virtual machine, we execute the apt-get command to install nagios3.

sudo apt-get update
sudo apt-get -y install nagios3

Nagios Event Brokers

Nagios Event Brokers (NEB) are modules hooked into the Nagios core during the startup process of Nagios. Event Brokers implement callback functions which are executed when special events occur in the Nagios server process.

I had the opportunity to work with 3 different Nagios event brokers :

• NDO, the well known broker used to send data to Centreon
• MKLivestatus, a very powerful broker that allows queries on Nagios’ data
• NTX module from Somone’s commerial solution TeeM Monitoring Hub

For this little project, I searched on Google a broker that could be use with Logstash. I found two interesting brokers on Github :

• https://github.com/mariussturm/nagios-zmq
• https://github.com/jbreams/nagmq

NagMQ’s development is more active and the project is also more production ready than the other. It is noteworthy that according to its README, Nagios-ZMQ is only a proof-of-concept. We will use this in this tutorial anyway because it’s easy to use and install.

Installation of Nagios-ZMQ

We have to install all the dependencies to build nagios-zmq.

sudo apt-get -y install libjson0-dev uuid-dev
sudo apt-get -y install libzmq-dev
git clone https://github.com/mariussturm/nagios-zmq.git
cd nagios-zmq/
make
sudo make install
make client

At the end of these steps, the file nagios-zmq.o is created in the directory /var/lib/nagios3.

We add our new broker in the Nagios configuration :

echo "broker_module=/var/lib/nagios3/nagios-zmq.o" >> /etc/nagios3/nagios.cfg
/etc/init.d/nagios3 restart

Restarting Nagios server produces the following output in the log :

[1411404964] message queue: nagios-zmq by Marius Sturm
[1411404964] message queue: successfully finished initialization
[1411404964] Event broker module '/var/lib/nagios3/nagios-zmq.o' initialized successfully.
[1411404964] Finished daemonizing... (New PID=22991)
[1411404964] message queue: start zmq publisher.
[1411404964] message queue: start zmq forwarder.

We can use the zmq_client shipped with nagios-zmq to check if the broker is working correctly :

vagrant@packer-virtualbox-iso:~/nagios-zmq$ ./zmq_client
Collecting updates from nagios server...
message: { "id": "7c126ae0-7104-404f-b57f-23e3984c7942", "context": "SERVICECHECK", "source": "NAGIOS", "timestamp": "1411405434", "payload": { "current_attempt": "1", "max_attempts": "4", "state_type": "1", "state": "0", "timestamp": "1411405434", "execution_time": "0.010529", "hostname": "localhost", "service": "HTTP", "output": "HTTP OK: HTTP\/1.1 200 OK - 454 bytes in 0.002 second response time", "performance": "time=0.002231s;;;0.000000 size=454B;;;0" } }
...

Installation of Logstash

We install Logstash using the official documentation.

sudo apt-get -y install default-jre
cd /opt
sudo curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar xvf logstash-1.4.2.tar.gz
cd logstash-1.4.2

We write a config file for Logstash with a zmq input and a stdout ouput to verify that our Logstash instance is able to subscribe to the ZMQ publisher.

input {
  zeromq {
    topology => "pubsub"
    address  => "tcp://127.0.0.1:6666"
    codec    => json {}
    mode     => "client"
  }
}
output {
  stdout {
    codec => rubydebug { }
  }
}

And it works :

vagrant@packer-virtualbox-iso:/opt/logstash-1.4.2$ ./bin/logstash agent -f agent.conf 
{
            "id" => "03b0181e-781c-44e3-ba28-22d2db1414f5",
       "context" => "HOSTCHECK",
        "source" => "NAGIOS",
     "timestamp" => "1411407454",
       "payload" => {
        "current_attempt" => "1",
           "max_attempts" => "10",
             "state_type" => "1",
                  "state" => "0",
              "timestamp" => "1411407454",
         "execution_time" => "0.004565",
               "hostname" => "localhost",
                 "output" => "PING OK - Packet loss = 0%, RTA = 0.04 ms",
            "performance" => "rta=0.038000ms;5000.000000;5000.000000;0.000000 pl=0%;100;100;0"
    },
      "@version" => "1",
    "@timestamp" => "2014-09-22T17:37:34.175Z",
          "host" => "packer-virtualbox-iso"
}
...