Monitoring Hadoop HDFS

Configure the Server Density agent to monitor your HDFS file system to:

• Collect important metrics, such as usage, replication & blocks
• Identify overall server slowdowns caused by HDFS

Monitored metrics

  • Date the last volume failure (epoch)
  • Remaining disk space left
  • Disk capacity
  • Disk used
  • Cache capacity
  • Cache used
  • Number of failed volumes
  • Estimated capacity lost
  • Number of blocks cached
  • Number of blocks that failed to cache
  • Number of blocks to remove from the cache
  • Disk usage
  • Disk usage percentage
  • Number of decommissioning live data nodes
  • Total number of live data nodes
  • Number of pending deletion blocks
  • Number of stale storages
  • Total number of dead data nodes
  • Number of decommissioning dead data nodes
  • Number of blocks scheduled for replication
  • Total number of blocks
  • Number of missing blocks
  • Number of decommissioning data nodes
  • Total number of files
  • Number of stale data nodes
  • Total volume failures
  • Total load on the file system
  • Remaining disk space left
  • Estimated capacity lost
  • Total disk capacity
  • Number of under replicated blocks
  • Number of corrupt blocks
  • Lock queue length
  • Maximum number of files HDFS supports
  • Number of blocks pending replication

Installing the HDFS plugin package

Install the HDFS plugins on Debian/Ubuntu:

sudo apt-get install sd-agent-hdfs

Install the HDFS plugin on RHEL/CentOS:

sudo yum install sd-agent-hdfs

Read more about agent plugins.

Configuring the agent to monitor HDFS

1. Configure /etc/sd-agent/conf.d/hdfs_datanode.yaml

init_config:

instances:
  # The HDFS DataNode check retrieves metrics from the HDFS DataNode's JMX
  # interface. This check must be installed on a HDFS DataNode. The HDFS
  # DataNode JMX URI is composed of the DataNode's hostname and port.
  #
  # The hostname and port can be found in the hdfs-site.xml conf file under
  # the property dfs.datanode.http.address
  #
  - hdfs_datanode_jmx_uri: http://localhost:50075

2. Configure /etc/sd-agent/conf.d/hdfs_namenode.yaml

init_config: 

instances:
  # The HDFS NameNode check retrieves metrics from the HDFS NameNode's JMX
  # interface. This check must be installed on the NameNode. The HDFS
  # NameNode JMX URI is composed of the NameNode's hostname and port.
  #
  # The hostname and port can be found in the hdfs-site.xml conf file under
  # the property dfs.http.address or dfs.namenode.http-address
  #
  - hdfs_namenode_jmx_uri: http://localhost:50070

3. Restart the agent

sudo /etc/init.d/sd-agent restart

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 

If the agent has been configured correctly you’ll see an output such as:

hdfs_datanode 
-------------  
- instance #0 [OK]  
- Collected 5 metrics & 1 service checks
hdfs_namenode 
-------------  
- instance #0 [OK]  
- Collected 4 metrics & 1 service checks

You can also check the metrics that the check is returning with the following commands:

sudo -u sd-agent /usr/share/python/sd-agent/agent.py hdfs_datanode
sudo -u sd-agent /usr/share/python/sd-agent/agent.py hdfs_namenode

Configuring graphs

Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the HDFS metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Monday  —  Friday.

10am  —  6pm UK.

Dedicated Support.