Steps to monitor Kafka & Storm:
– open JMX port for kafka and storm
– To open JMX port in Kafka :
a. Add below line to $KAFKA_HOME/bin/kafka-run-class.sh
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=xxx.xxx"
b. Add below line to $KAFKA_HOME/bin/kafka-server-start.sh
export JMX_PORT=${JMX_PORT:-2999}
c. Restart kafka
$KAFKA_HOME/bin/kafka-server-stop.sh $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
– To open JMX port in Storm:
a. Add below lines $STORM_HOME/conf/storm.yaml
nimbus.childopts: "-Xmx1024m -Dcom.sun.management.jxmremote -Dcom.sun.management.jmxremote.local.only=false - Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=2998 -Djava.rmi.server.hostname=xxx.xxx" supervisor.childopts: "-Xmx1024m -Dcom.sun.management.jxmremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=2997 -Djava.rmi.server.hostname=xxx.xxx"
b. Restart Storm
for i in `ps -ef|grep storm|grep -v grep|awk '{print $2}'`; do kill -9 $i; done
$STORM_HOME/bin/storm nimbus &
$STORM_HOME/bin/storm supervisor &
$STORM_HOME/bin/storm ui &
$STORM_HOME/bin/storm jar <topology jar file> Main &
– Check ports opened or not
netstat -antl|grep 299
You should see 2999, 2998 & 2997 ports opened.
– Download check_jmx plugin from https://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jmx/details or
wget -c http://snippets.syabru.ch/nagios-jmx-plugin/download/nagios-jmx-plugin.zip
To use check_jmx plugin you need to have java installed on your server.
– Extract downloaded archive
– copy check_jmx & check_jmx.jar to your nagios plugins folder (Usually under “/usr/lib64/nagios/plugins/ ”
– You can check various arguments by running /usr/lib64/nagios/plugins/check_jmx –help
– Configure command to check kafka and storm in Icinga2
Add below lines to /etc/icinga2/conf.d/commands.conf
object CheckCommand "check_jmx" {
import "plugin-check-command"
command = [ PluginDir + "/check_jmx" ]
arguments = {
"-U" = "$service_url$"
"-O" = "$object_name$"
"-A" = "$attrib_name$"
"-K" = "$comp_key$"
"-w" = "$warn$"
"-c" = "$crit$"
"-o" = "$operat_name$"
"--username" = "$username$"
"--password" = "$password$"
"-u" = "$unit$"
"-v" = "$verbose$"
}
}
– Configure Services to check kafka and storm in Icinga2
Add below lines to /etc/icinga2/conf.d/services.conf
apply Service "kafka" {
import "generic-service"
check_command = "check_jmx"
vars.service_url = "service:jmx:rmi:///jndi/rmi://" + host.name + ":2997/jmxrmi"
vars.object_name = "java.lang:type=Memory"
vars.attrib_name = "HeapMemoryUsage"
vars.comp_key = "used"
vars.warn = "838860800"
vars.crit = "996147200"
assign where match("xxx.xxx", host.name)
}
apply Service "stormN" {
import "generic-service"
check_command = "check_jmx"
vars.service_url = "service:jmx:rmi:///jndi/rmi://" + host.name + ":2998/jmxrmi"
vars.object_name = "java.lang:type=Memory"
vars.attrib_name = "HeapMemoryUsage"
vars.comp_key = "used"
vars.warn = "838860800"
vars.crit = "996147200"
assign where match("xxx.xxx", host.name)
}
apply Service "stormS" {
import "generic-service"
check_command = "check_jmx"
vars.service_url = "service:jmx:rmi:///jndi/rmi://" + host.name + ":2997/jmxrmi"
vars.object_name = "java.lang:type=Memory"
vars.attrib_name = "HeapMemoryUsage"
vars.comp_key = "used"
vars.warn = "838860800"
vars.crit = "996147200"
assign where match("xxx.xxx", host.name)
}
Steps to monitor Zookeeper:
– You must have Python 2.6+ installed
– Download check_zookeeper.py plugin for Nagios and place it under Nagios Plugin directory such /usr/lib64/nagios/plugins/
wget -c http://svn.apache.org/repos/asf/zookeeper/trunk/src/contrib/monitoring/check_zookeeper.py
You can check multiple options of check_zookeepr.py by using below command:
check_zookeeper.py –help
– Configure command to icinga2 configuration:
Add below lines to /etc/icinga2/conf.d/commands.conf
object CheckCommand "check_zoo" {
import "plugin-check-command"
command = [ PluginDir + "/check_zookeeper.py" ]
arguments = {
"-o" = "$mon_tool$"
"-s" = "$server$"
"-k" = "$key$"
"-w" = "$warn$"
"-c" = "$crit$"
}
}
– Configure Service to icinga2 configuration:
apply Service "Zoo" {
import "generic-service"
check_command = "check_zoo"
vars.mon_tool= "nagios"
vars.server= host.name + ":2181"
vars.key = "zk_avg_latency"
vars.comp_key = "used"
vars.warn = "7"
vars.crit = "14"
assign where match("xxx.xxx", host.name)
}
To make this working you should have entry for your xxx.xxx host in icinga2. I have added below entry to /etc/icinga2/conf.d/hosts.conf
object Host "xxx.xxx" {
import "generic-host"
address = "NNN.NNN.NNN.NNN"
}
– reload icinga2 configuration
# /etc/init.d/icinga2 reload
Now you can login to icingaweb2 web console and check.
Please note:
* I have configured Icinga2 to monitor HeapMemorySize of Storm Nimbus, Supervisor and Kafka. You can monitor any other JVM parameter which you can access using JMX.
* In above example for kafka and storm my maximum heapsize is 1024 MB. So I have configured Warning level at 800MB and Critical at 950MB. You can configure it by your requirements.
* For zookeeper I have configured monitor for zk_avg_latency. You can monitor any other parameter as per your requirements.
* you can use any service & command name instead of check_jmx, stormN, stormS zoo.
* I have used CentOS6.5 as base OS for icinga2, zookeeper, kafka & storm.
