Monitoring Kafka Consumer

This article will help you get the Kafka_consumer plugin for sd-agent configured and returning metrics

Installing the kafka_consumer plugin package

Install the kafka_consumer plugin on Debian/Ubuntu:

sudo apt-get install sd-agent-kafka_consumer

Install the kafka_consumer plugin on RHEL/CentOS:

sudo yum install sd-agent-kafka_consumer

Read more about agent plugins.

Configuring the agent to monitor Apache Kafka consumer

1. The guide below is for Kafka >= 0.8.2.

2. Configure /etc/sd-agent/conf.d/kafka_consumer.yaml

init_config:
#  Customize the ZooKeeper connection timeout here
#  zk_timeout: 5
#  Customize the Kafka connection timeout here
#  kafka_timeout: 5

instances:
   - kafka_connect_str: localhost:19092
     zk_connect_str: localhost:2181
     zk_prefix: /0.8
     consumer_groups:
       my_consumer:
         my_topic: [0, 1, 4, 12]

3. Restart the agent

sudo /etc/init.d/sd-agent restart

or

sudo systemctl restart sd-agent

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 

or

/usr/share/python/sd-agent/agent.py info

If the agent has been configured correctly you'll see an output such as:

kafka_consumer
-----
  - instance #0 [OK]
  - Collected * metrics

You can also view the metrics returned with the following command:

sudo -u sd-agent /usr/share/python/sd-agent/agent.py check kafka_consumer

Configuring graphs

Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the kafka_consumer metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

Monitored metrics

Metric	Values
kafka.broker_offset Current message offset on broker.	offset / None Type: float
kafka.consumer_lag Lag in messages between consumer and broker.	offset / None Type: float
kafka.consumer_offset Current message offset on consumer.	offset / None Type: float