Monitoring JMX/JVM – Server Density

Configure the Server Density agent to monitor your Jvm instance to:

• Collect important Jvm metrics
• Identify overall service slowdowns caused by your Jvm instances

Installing the jvm plugin package

Install the jvm plugin on Debian/Ubuntu:

sudo apt-get install sd-agent-jmx

Install the jvm plugin on RHEL/CentOS:

sudo yum install sd-agent-jmx

Read more about agent plugins.

Configuring the agent to monitor JMX metrics

1. Configure /etc/sd-agent/conf.d/jmx.yaml

The JMX check automatically collects 11 metrics. You must configure the plugin to collect other metrics if they are required.

init_config:
  custom_jar_paths: # optional
    - /path/to/custom/jarfile.jar

instances:
  - host: localhost
    port: 7199
    user: username
    password: password

    jmx_url: "service:jmx:rmi:///jndi/rmi://myhost.host:9999/custompath" # optional

    name: jmx_instance  # optional
    java_bin_path: /path/to/java
    java_options: "-Xmx200m -Xms50m"
    trust_store_path: /path/to/trustStore.jks
    trust_store_password: password

    process_name_regex: .*process_name.*
    tools_jar_path: /usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar
    refresh_beans: 600 # optional (in seconds)
    tags:
      env: stage
      newTag: test

    conf:
      - include:
          domain: my_domain
          tags:
              simple: $attr0
              raw_value: my_chosen_value
              multiple: $attr0-$attr1
          bean:
            - my_bean
            - another_bean
          attribute:
            attribute1:
              metric_type: counter
              alias: jmx.my_metric_name
            attribute2:
              metric_type: gauge
              alias: jmx.my2ndattribute
      - include:
          domain: 2nd_domain
        exclude:
          bean:
            - excluded_bean
      - include:
          domain_regex: regex_on_domain
        exclude:
          bean_regex:
            - regex_on_excluded_bean

Configuration Options

custom_jar_paths (optional) - This allows you to specify custom jars that will be added the agent's JVM classpath.
jmx_url (optional) - You can use this to specify if the agent needs to connect to a JMX URL other than the default. Name must also be specified if this option is configured.
name (optional) - Used when jmx_url is configured.
java_bin_path (optional) - Set this if sd-agent is unable to find your java executable.
java_options (optional) - Allows you to specify Java JVM options
trust_store_path (optional) - Must be set if SSL is enabled.
trust_store_password (optional) - Must be set if SSL is enabled.
process_name_regex (optional) - Use this to request sd-agent connect using the attach api. This requires the JDK to be installed and tools_jar_path be set to the path of tools.jar
tools_jar_path (optional) - This should be set if process_name_regex is configured, to the path of tools.jar
refresh_beans (optional) - The period for refreshing the matching MBeans list. This defaults to 600 seconds. Decreasing this may result in increased CPU usage
conf - Is a list of dictionaries in which only 2 keys are allowed, include and exclude
- include (mandatory) - A dictionary of filters. If an attribute matches this filter it will be collected as metric, unless it is also matched by the exclude filter
- exclude (optional) - A dictionary of filters If an attribute matches this filter it will not be collected as a metric, even if it matches the include filter.

Filter Descriptions

The include and exclude dictionaries can support the following keys:

domain - A list of domain names such as java.lang
domain_regex - A list of regexes for domain names. Must conform to Java regex
bean or bean_name - A list of full bean names such as java.lang:type=Compilation
bean_regex - A list of regexes for bean_names. Must conform to Java regex
attribute - A list or dictionary of attribute names

Attribute Filters

The attribute filter will accept a dictionary where the keys are attribute names, or a list of a attribute names. For example, a dictionary where the keys are attribute names could be formed as:

conf:
  - include:
      attribute:
        maxThreads:
          alias: tomcat.threads.max
          metric_type: gauge
        currentThreadCount:
          alias: tomcat.threads.count
          metric_type: gauge
        bytesReceived:
          alias: tomcat.bytes_rcvd
          metric_type: counter

As an alias is specified, this is the name that will be displayed for the metric in your Server Density account. You can also specify the metric type as either a gauge or a counter. The agent will automatically compute a rate per second for this metric if you specify the metric as a counter.

An example of a list of attribute names:

conf:
  - include:
      domain: org.apache.cassandra.db
      attribute:
        - BloomFilterDiskSpaceUsed
        - BloomFilterFalsePositives
        - BloomFilterFalseRatio
        - Capacity
        - CompressionRatio
        - CompletedTasks
        - ExceptionCount
        - Hits
        - RecentHitRate

With this example the metric types will all be set as gauge and the metric names will be jmx.$DOMAIN_NAME.$ATTRIBUTE_NAME.

Troubleshooting

You can use the sd-agent jmx command to help you troubleshoot your JMX/JVM plugin.

sudo /etc/init.d/sd-agent jmx list_matching_attributes - Lists the attributes that match your instance configurations.
sudo /etc/init.d/sd-agent jmx list_limited_attributes - List attributes that match your instance configuration but that aren't being collected due to exceeding the metric limit.
sudo /etc/init.d/sd-agent jmx list_collected_attributes - Lists the attributes that will be collected by your current instance configurations.
sudo /etc/init.d/sd-agent jmx list_not_matching_attributes - List attributes that do not match any of your current instance configurations.
sudo /etc/init.d/sd-agent jmx list_everything - List every attribute that could be collected.
sudo /etc/init.d/sd-agent jmx collect - Start metric collection and display them in the console.

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info

If the agent has been configured correctly you ll see an output such as:

jvm
-----
  - instance #0 [OK]
  - Collected * metrics & * events

You can also view the metrics returned with the following command:

sudo -u sd-agent /usr/share/python/sd-agent/agent.py check jvm

Configuring graphs

Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the jvm metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

Monitored metrics

Metric	Values
jvm.heap_memory The amount of Java heap memory used.	byte / None Type: float
jvm.heap_memory_committed The total amount of Java heap memory commited to be used	byte / None Type: float
jvm.heap_memory_init The initial amount of Java heap memory allocated.	byte / None Type: float
jvm.heap_memory_max The maximum amount of Java heap memory available.	byte / None Type: float
jvm.non_heap_memory The total amount of Java non-heap memory used.	byte / None Type: float
jvm.non_heap_memory_committed The total amount of Java non-heap memory committed to be used..	byte / None Type: float
jvm.non_heap_memory_init The initial Java non-heap memory allocated.	byte / None Type: float
jvm.non_heap_memory_max The maximum Java non-heap memory available.	byte / None Type: float
jvm.thread_count The number of active threads in the JVM.	thread / None Type: float
jvm.gc.cms.count The total number of garbage collections run by the JVM.	garbage collection / None Type: float
jvm.gc.parnew.time The approximate accumulated garbage collection time elapsed.	millisecond / None Type: float

Comments