Configure the Server Density agent to monitor your Apache Kafka consumer client to:
• Collect important metrics to graph and monitor performance
• Identify overall server slowdowns caused by the cluster
The Apache Kafka consumer plugin allows you to monitor the following metrics for each topic and partition:
- Broker offset
- Consumer lag
- Consumer offset
Installing the Kafka Consumer plugin package
Install the Kafka Consumer plugin on Debian/Ubuntu:
sudo apt-get install sd-agent-kafka-consumer
Install the Kafka Consumer plugin on RHEL/CentOS:
sudo yum install sd-agent-kafka-consumer
Read more about agent plugins.
Configuring the agent to monitor Apache Kafka consumer
1. The guide below is for Kafka >= 0.8.2.
2. Configure /etc/sd-agent/conf.d/kafka_consumer.yaml
init_config: # Customize the ZooKeeper connection timeout here # zk_timeout: 5 # Customize the Kafka connection timeout here # kafka_timeout: 5 instances: - kafka_connect_str: localhost:19092 zk_connect_str: localhost:2181 zk_prefix: /0.8 consumer_groups: my_consumer: my_topic: [0, 1, 4, 12]
3. Restart the agent
sudo /etc/init.d/sd-agent restart
Verifying the configuration
Execute info to verify the configuration with the following:
sudo /etc/init.d/sd-agent info
If the agent has been configured correctly you’ll see an output such as:
- instance #0 [OK]
- Collected 8 metrics & 0 events
You can also view the metrics returned with the following command:
sudo -u sd-agent /usr/share/python/sd-agent/agent.py check kafka_consumer
Metrics will be in the form
$topic_$partition so for example a topic named
cloud with 10 partitions would need 10 alerts, e.g.
Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the Apache Kafka metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.