Configure the Server Density agent to monitor your Apache Kafka cluster to:
• Collect important metrics to graph and monitor performance
• Identify overall server slowdowns caused by the cluster
Note: This plugin is still in development and is not yet available to install. We will add new plugins to our release notes.
The Apache Kafka plugin is limited to 350 metrics per cluster/instance. You can configure the metrics to be returned using the config files (see below), or read more about JMX Checks.
Configuring the agent to monitor Apache Kafka
1. The guide below is for Kafka >= 0.8.2.
2. Configure /etc/sd-agent/conf.d/kafka.yaml
instances: # - host: localhost # port: 9999 # name: jmx_instance # user: username # password: password # #java_bin_path: /path/to/java #Optional # #trust_store_path: /path/to/trustStore.jks # Optional # #trust_store_password: password
3. Configure /etc/sd-agent/conf.d/kafka_consumer.yaml
init_config: # Customize the ZooKeeper connection timeout here # zk_timeout: 5 # Customize the Kafka connection timeout here # kafka_timeout: 5 instances: # - kafka_connect_str: localhost:19092 # zk_connect_str: localhost:2181 # zk_prefix: /0.8 # consumer_groups: # my_consumer: # my_topic: [0, 1, 4, 12]
4. Restart the agent
sudo /etc/init.d/sd-agent restart
Verifying the configuration
Execute info to verify the configuration with the following:
sudo /etc/init.d/sd-agent info
If the agent has been configured correctly you’ll see an output such as:
- instance #0 [OK]
- Collected 8 metrics & 0 events
Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the Apache Kafka metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.
We have a detailed blog post about how to monitor kafka.