Monitoring HDFS Datanode

This article will help you get the Hdfs_datanode plugin for sd-agent configured and returning metrics

Installing the hdfs_datanode plugin package

Install the hdfs_datanode plugin on Debian/Ubuntu:

sudo apt-get install sd-agent-hdfs_datanode

Install the hdfs_datanode plugin on RHEL/CentOS:

sudo yum install sd-agent-hdfs_datanode

Read more about agent plugins.

Configuring the agent to monitor HDFS

1. Configure /etc/sd-agent/conf.d/hdfs_datanode.yaml


# The HDFS DataNode check retrieves metrics from the HDFS DataNode's JMX
# interface. This check must be installed on a HDFS DataNode. The HDFS
# DataNode JMX URI is composed of the DataNode's hostname and port.
# The hostname and port can be found in the hdfs-site.xml conf file under
# the property dfs.datanode.http.address
- hdfs_datanode_jmx_uri: http://localhost:50075

2. Restart the agent

sudo /etc/init.d/sd-agent restart


sudo systemctl restart sd-agent

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 


/usr/share/python/sd-agent/ info

If the agent has been configured correctly you'll see an output such as:

  - instance #0 [OK]
  - Collected * metrics

You can also view the metrics returned with the following command:

sudo -u sd-agent /usr/share/python/sd-agent/ check hdfs_datanode

Configuring graphs

Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the hdfs_datanode metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.


Monitored metrics

Metric Values

Cache capacity in bytes
byte / None
Type: float

Cache used in bytes
byte / None
Type: float

Disk capacity in bytes
byte / None
Type: float

The remaining disk space left in bytes
byte / None
Type: float

Disk usage in bytes
byte / None
Type: float

The estimated capacity lost in bytes
byte / None
Type: float

The date/time of the last volume failure in milliseconds since epoch
millisecond / None
Type: float

The number of blocks cached
block / None
Type: float

The number of blocks that failed to cache
block / None
Type: float

The number of failed blocks to remove from cache
block / None
Type: float

Number of failed volumes
None / None
Type: float
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Monday  —  Friday.

10am  —  6pm UK.

Dedicated Support.