Monitoring Apache Kafka consumer

Configure the Server Density agent to monitor your Apache Kafka consumer client to:

• Collect important metrics to graph and monitor performance
• Identify overall server slowdowns caused by the cluster

The Apache Kafka consumer plugin allows you to monitor the following metrics for each topic and partition:

  • Broker offset
  • Consumer lag
  • Consumer offset

Installing the Kafka Consumer plugin package

Install the Kafka Consumer plugin on Debian/Ubuntu:

sudo apt-get install sd-agent-kafka-consumer

Install the Kafka Consumer plugin on RHEL/CentOS:

sudo yum install sd-agent-kafka-consumer

Read more about agent plugins.

Configuring the agent to monitor Apache Kafka consumer

1. The guide below is for Kafka >= 0.8.2.

2. Configure /etc/sd-agent/conf.d/kafka_consumer.yaml

init_config:
#  Customize the ZooKeeper connection timeout here
#  zk_timeout: 5
#  Customize the Kafka connection timeout here
#  kafka_timeout: 5

instances:
   - kafka_connect_str: localhost:19092
     zk_connect_str: localhost:2181
     zk_prefix: /0.8
     consumer_groups:
       my_consumer:
         my_topic: [0, 1, 4, 12]

3. Restart the agent

sudo /etc/init.d/sd-agent restart

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 

If the agent has been configured correctly you’ll see an output such as:

kafka-localhost-9999
-----
- instance #0 [OK]
- Collected 8 metrics & 0 events

You can also view the metrics returned with the following command:

sudo -u sd-agent /usr/share/python/sd-agent/agent.py check kafka_consumer

Configuring Alerts
Metrics will be in the form $topic_$partition so for example a topic named cloud with 10 partitions would need 10 alerts, e.g. cloud_1, cloud_2 etc 

Configuring Graphs
Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the Apache Kafka metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Monday  —  Friday.

10am  —  6pm UK.

Dedicated Support.