[In Development] Monitoring Apache Kafka

Configure the Server Density agent to monitor your Apache Kafka cluster to:

• Collect important metrics to graph and monitor performance
• Identify overall server slowdowns caused by the cluster

Note: This plugin is still in development and is not yet available to install. We will add new plugins to our release notes.

The Apache Kafka plugin is limited to 350 metrics per cluster/instance. You can configure the metrics to be returned using the config files (see below), or read more about JMX Checks.

Configuring the agent to monitor Apache Kafka

1. The guide below is for Kafka >= 0.8.2.

2. Configure /etc/sd-agent/conf.d/kafka.yaml

instances:
#    -   host: localhost
#        port: 9999
#        name: jmx_instance
#        user: username
#        password: password
#        #java_bin_path: /path/to/java #Optional
#        #trust_store_path: /path/to/trustStore.jks # Optional
#        #trust_store_password: password

3. Configure /etc/sd-agent/conf.d/kafka_consumer.yaml

init_config:
#  Customize the ZooKeeper connection timeout here
#  zk_timeout: 5
#  Customize the Kafka connection timeout here
#  kafka_timeout: 5

instances:
  # - kafka_connect_str: localhost:19092
  #   zk_connect_str: localhost:2181
  #   zk_prefix: /0.8
  #   consumer_groups:
  #     my_consumer:
  #       my_topic: [0, 1, 4, 12]

4. Restart the agent

sudo /etc/init.d/sd-agent restart

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 

If the agent has been configured correctly you’ll see an output such as:

kafka-localhost-9999
-----
- instance #0 [OK]
- Collected 8 metrics & 0 events

Configuring Graphs
Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the Apache Kafka metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

Further reading

We have a detailed blog post about how to monitor kafka.

Was this article helpful?
0 out of 1 found this helpful
Have more questions? Submit a request

Comments

Monday  —  Friday.

10am  —  6pm UK.

Dedicated Support.