Send Zabbix events to Logstash

Zabbix is an enterprise-class open source distributed monitoring solution for networks and applications and Logstash is a tool for managing events and logs.

Logstash has already a Zabbix output to send data from Logstash. It can be interesting to send Zabbix data to Logstash to maintain a highly decoupled IT monitoring infrastructure.

By sending Zabbix events to Logstash, one can easily integrate Zabbix with a lot of things. This includes :

• Integrate Zabbix with Pagerdudy, JIRA or redmine…
• Integrate new medias : Send Alerts to a XMPP conference room, to a IRC channel, to mobile device applications using Amazon SNS…
• Send events to another application via TCP, Amazon SQS, websocket, rabbitmq or zeromq…
• Backup / Store Zabbix alerts in Excel files, Amazon S3, mongodb or elasticsearch…

I already explained in a previous post how to get data from Zabbix. Today, we will detail how to use an alert script to send Zabbix events in real time to Logstash.

How we will integrate Zabbix with Logstash

In short, we will add a Zabbix user with read only permissions. This user will have only one media which is just a script (an alert script). Then, we will configure Zabbix to notify this user with a special message if any event occur.

The special message will contain all the macros available for an alert script (List available here, first column for our case). This special message will be formated in INI style.

The alert script I made in python will parse the message in INI format and convert it to JSON before sending it to Logstash using a TCP port.

Step by step integration tutorial

• Step 1 : Download my python script zabbix2logstash.py into /etc/zabbix/alert.d/ :

wget https://gist.githubusercontent.com/plewin/2808ddbd32bcb756e5ba84da51b2ad6a/raw/a939d59ad4f86ae60b4e4b8dda90b728ae25289a/zabbix2logstash.py -O /etc/zabbix/alert.d/zabbix2logstash.py

• Step 2 : Set the script executable :

chmod +x /etc/zabbix/alert.d/zabbix2logstash.py

• Step 3 : Click on Media types from Administration tab.
• Step 4 : Click on Create Media Type button.

• Step 5 : Enter Logstash for the name.
• Step 6 : Set Type to Script.
• Step 7 : Enter zabbix2logstash.py as the Script name.
• Step 8 : Click Save to save the Logstash media type.

• Step 9 : Click on Users from Administration tab.
• Step 10 : Select Users groups from the drop-down list.
• Step 11 : Click on Create user group button.

• Step 12 : Enter Logstash Group as the group name.
• Step 13 : Set Frontend access to Disabled.

• Step 14 : Click on Permissions tab.
• Step 15 : Click on Add and everything in the read only category.
• Step 16 : Click on Save to save the new group.

• Step 17 : Click on Users from Administration tab.
• Step 18 : Select Users from the drop-down list.
• Step 19 : Click on Create user button.

• Step 20 : Enter Logstash User as alias.
• Step 21 : Enter Logstash User as name.
• Step 22 : Enter logstash as surname.
• Step 23 : Add the user to the Logstash Group.
• Step 24 : Set a difficult password (possibly anything random, we will not use the password).

• Step 25 : Click on Media tab.
• Step 26 : Click on Add button.

• Step 27 : Select Logstash from the drop-down list.
• Step 28 : Enter Logstash’s hostname or IP address in the send to field.
• Step 29 : Click on Add button.

• Step 30 : Click on Save.

• Step 31 : Click on Actions from Configuration tab.
• Step 32 : Click on Create action.

• Step 33 : Enter Logstash Notification as the name.
• Step 34 : Enter Logstash’s tcp port in the default subject.
• Step 35 : Copy paste the ini template and add message_type=default under [zabbix_data].
• Step 36 : Check Recovery Message.
• Step 37 : Enter Logstash’s tcp port in the recovery subject.
• Step 38 : Copy paste the ini template and add message_type=recovery under [zabbix_data].

• Step 39 : Click on Conditions tab.
• Step 40 : Remove the second condition to keep only the condition “Maintenance status not in maintenance”.

• Step 41 : Click on Operations tab.
• Step 42 : Click on New.

• Step 43 : In the Send to Users list, click on Add and add Logstash User to the list.
• Step 44 : Click on Add at the bottom of the page to validate the new operation.
• Step 45 : Click on Save at the bottom of the page to save our new action.

Done !

Logstash configuration

This Logstash configuration will receive events from Zabbix and will print all events in the standard ouput.

input {
  tcp {
    port => 5100
  }
}

filter {
  json {
    source => "message"
    remove_field => "message"
  }
}

output {
  stdout {
    codec => rubydebug { }
  }
  
}

Resources

Files

• Sample Zabbix Data Dump of a Zabbix event sent to logstash
• Zabbix Message All Zabbix macros available in INI format
• zabbix2logstash.py Alertscript to send Zabbix events to logstash using tcp and json