This article will help you get the Elasticsearch plugin for sd-agent configured and returning metrics
Installing the elasticsearch plugin package
Install the elasticsearch plugin on Debian/Ubuntu:
sudo apt-get install sd-agent-elastic
Install the elasticsearch plugin on RHEL/CentOS:
sudo yum install sd-agent-elastic
Read more about agent plugins.
Configuring the agent to monitor Elastic
1. Configure /etc/sd-agent/conf.d/elastic.yaml
init_config:
instances:
- url: http://localhost:9200
- If Elastic requires authentication or you wish to specify other connection details, amend the rest of the config file as necessary
2. Restart the agent
sudo /etc/init.d/sd-agent restart
or
sudo systemctl restart sd-agent
Verifying the configuration
Execute info to verify the configuration with the following:
sudo /etc/init.d/sd-agent info
or
/usr/share/python/sd-agent/agent.py info
If the agent has been configured correctly you'll see an output such as:
elasticsearch ----- - instance #0 [OK] - Collected * metrics
You can also view the metrics returned with the following command:
sudo -u sd-agent /usr/share/python/sd-agent/agent.py check elastic
Configuring graphs
Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the elasticsearch metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.
Monitored metrics
| Metric | Values |
|---|---|
| elasticsearch.active_primary_shards |
/ Type: float |
| elasticsearch.active_shards |
/ Type: float |
| elasticsearch.breakers.fielddata.estimated_size_in_bytes The estimated size in bytes of the field data circuit breaker. |
byte / None Type: float |
| elasticsearch.breakers.fielddata.overhead The constant multiplier for byte estimations of the field data circuit breaker. |
None / None Type: float |
| elasticsearch.breakers.fielddata.tripped The number of times the field data circuit breaker has tripped. |
None / None Type: float |
| elasticsearch.breakers.parent.estimated_size_in_bytes The estimated size in bytes of the parent circuit breaker. |
byte / None Type: float |
| elasticsearch.breakers.parent.overhead The constant multiplier for byte estimations of the parent circuit breaker. |
None / None Type: float |
| elasticsearch.breakers.parent.tripped The number of times the parent circuit breaker has tripped. |
None / None Type: float |
| elasticsearch.breakers.request.estimated_size_in_bytes The estimated size in bytes of the request circuit breaker. |
byte / None Type: float |
| elasticsearch.breakers.request.overhead The constant multiplier for byte estimations of the request circuit breaker. |
None / None Type: float |
| elasticsearch.breakers.request.tripped The number of times the request circuit breaker has tripped. |
None / None Type: float |
| elasticsearch.cache.field.evictions The total number of evictions from the field data cache. |
eviction / None Type: float |
| elasticsearch.cache.field.size The size of the field cache. |
byte / None Type: float |
| elasticsearch.cache.filter.count The number of items in the filter cache. |
item / None Type: float |
| elasticsearch.cache.filter.evictions The total number of evictions from the filter cache. |
eviction / None Type: float |
| elasticsearch.cache.filter.size The size of the filter cache. |
byte / None Type: float |
| elasticsearch.cluster_status |
/ Type: float |
| elasticsearch.docs.count |
/ Type: float |
| elasticsearch.docs.deleted |
/ Type: float |
| elasticsearch.fielddata.evictions |
/ Type: float |
| elasticsearch.fielddata.size |
byte / Type: float |
| elasticsearch.flush.total |
/ Type: float |
| elasticsearch.flush.total.time |
/ Type: float |
| elasticsearch.fs.total.available_in_bytes |
byte / Type: float |
| elasticsearch.fs.total.disk_io_op The total I/O operations on the file store. |
operation / None Type: float |
| elasticsearch.fs.total.disk_io_size_in_bytes Total bytes used for all I/O operations on the file store. |
byte / None Type: float |
| elasticsearch.fs.total.disk_read_size_in_bytes The total bytes read from the file store. |
byte / None Type: float |
| elasticsearch.fs.total.disk_reads The total number of reads from the file store. |
read / None Type: float |
| elasticsearch.fs.total.disk_write_size_in_bytes The total bytes written to the file store. |
byte / None Type: float |
| elasticsearch.fs.total.disk_writes The total number of writes to the file store. |
write / None Type: float |
| elasticsearch.fs.total.free_in_bytes |
byte / Type: float |
| elasticsearch.fs.total.total_in_bytes |
byte / Type: float |
| elasticsearch.get.active |
/ Type: float |
| elasticsearch.get.current |
/ Type: float |
| elasticsearch.get.exists.time |
second / Type: float |
| elasticsearch.get.exists.total |
/ Type: float |
| elasticsearch.get.missing.time |
second / Type: float |
| elasticsearch.get.missing.total |
/ Type: float |
| elasticsearch.get.queue |
/ Type: float |
| elasticsearch.get.threads |
/ Type: float |
| elasticsearch.get.time |
second / Type: float |
| elasticsearch.get.total |
/ Type: float |
| elasticsearch.http.current_open |
/ Type: float |
| elasticsearch.http.total_opened |
/ Type: float |
| elasticsearch.id_cache.size The size of the id cache |
byte / None Type: float |
| elasticsearch.indexing.delete.current |
/ Type: float |
| elasticsearch.indexing.delete.time |
second / Type: float |
| elasticsearch.indexing.delete.total |
/ Type: float |
| elasticsearch.indexing.index.current |
/ Type: float |
| elasticsearch.indexing.index.time |
second / Type: float |
| elasticsearch.indexing.index.total |
/ Type: float |
| elasticsearch.indices.indexing.index_failed The number of failed indexing operations. |
None / None Type: float |
| elasticsearch.indices.indexing.throttle_time The total time indexing waited due to throttling. |
millisecond / None Type: float |
| elasticsearch.indices.query_cache.evictions The number of query cache evictions. |
eviction / None Type: float |
| elasticsearch.indices.query_cache.hit_count The number of query cache hits. |
hit / None Type: float |
| elasticsearch.indices.query_cache.memory_size_in_bytes The memory used by the query cache. |
byte / None Type: float |
| elasticsearch.indices.query_cache.miss_count The number of query cache misses. |
miss / None Type: float |
| elasticsearch.indices.recovery.current_as_source The number of ongoing recoveries for which a shard serves as a source. |
None / None Type: float |
| elasticsearch.indices.recovery.current_as_target The number of ongoing recoveries for which a shard serves as a target. |
None / None Type: float |
| elasticsearch.indices.recovery.throttle_time The total time recoveries waited due to throttling. |
millisecond / None Type: float |
| elasticsearch.indices.request_cache.evictions The number of request cache evictions. |
eviction / None Type: float |
| elasticsearch.indices.request_cache.hit_count The number of request cache hits. |
hit / None Type: float |
| elasticsearch.indices.request_cache.memory_size_in_bytes The memory used by the request cache. |
byte / None Type: float |
| elasticsearch.indices.request_cache.miss_count The number of request cache misses. |
miss / None Type: float |
| elasticsearch.indices.segments.count |
/ Type: float |
| elasticsearch.indices.segments.doc_values_memory_in_bytes The memory used by doc values. |
byte / None Type: float |
| elasticsearch.indices.segments.fixed_bit_set_memory_in_bytes |
/ Type: float |
| elasticsearch.indices.segments.index_writer_max_memory_in_bytes |
/ Type: float |
| elasticsearch.indices.segments.index_writer_memory_in_bytes |
/ Type: float |
| elasticsearch.indices.segments.memory_in_bytes |
byte / Type: float |
| elasticsearch.indices.segments.norms_memory_in_bytes The memory used by norms. |
byte / None Type: float |
| elasticsearch.indices.segments.stored_fields_memory_in_bytes The memory used by stored fields. |
byte / None Type: float |
| elasticsearch.indices.segments.term_vectors_memory_in_bytes The memory used by term vectors. |
byte / None Type: float |
| elasticsearch.indices.segments.terms_memory_in_bytes The memory used by terms. |
byte / None Type: float |
| elasticsearch.indices.segments.version_map_memory_in_bytes |
byte / Type: float |
| elasticsearch.indices.translog.operations |
/ Type: float |
| elasticsearch.indices.translog.size_in_bytes |
byte / Type: float |
| elasticsearch.initializing_shards |
/ Type: float |
| elasticsearch.merges.current |
/ Type: float |
| elasticsearch.merges.current.docs |
/ Type: float |
| elasticsearch.merges.current.size |
byte / Type: float |
| elasticsearch.merges.total |
/ Type: float |
| elasticsearch.merges.total.docs |
/ Type: float |
| elasticsearch.merges.total.size |
/ Type: float |
| elasticsearch.merges.total.time |
second / Type: float |
| elasticsearch.number_of_data_nodes |
/ Type: float |
| elasticsearch.number_of_nodes |
/ Type: float |
| elasticsearch.pending_tasks_priority_high |
/ Type: float |
| elasticsearch.pending_tasks_priority_urgent |
/ Type: float |
| elasticsearch.pending_tasks_total |
/ Type: float |
| elasticsearch.primaries.docs.count The total number of documents in the primary shards. |
document / None Type: float |
| elasticsearch.primaries.docs.deleted The total number of documents deleted from the primary shards. |
document / None Type: float |
| elasticsearch.primaries.flush.total The total number of index flushes to disk from the primary shards since start. |
flush / None Type: float |
| elasticsearch.primaries.flush.total.time The total time spent flushing the index to disk from the primary shards. |
second / None Type: float |
| elasticsearch.primaries.get.current The number of get requests currently running on the primary shards. |
request / None Type: float |
| elasticsearch.primaries.get.exists.time The total time spent on get requests from the primary shards where the document existed. |
request / None Type: float |
| elasticsearch.primaries.get.exists.total The total number of get requests on primary shards where the document existed. |
request / None Type: float |
| elasticsearch.primaries.get.missing.time The total time spent on get requests from the primary shards where the document was missing. |
second / None Type: float |
| elasticsearch.primaries.get.missing.total The total number of get requests from the primary shards where the document was missing. |
request / None Type: float |
| elasticsearch.primaries.get.time The total time spent on get requests from the primary shards. |
second / None Type: float |
| elasticsearch.primaries.get.total The total number of get requests from the primary shards. |
request / None Type: float |
| elasticsearch.primaries.indexing.delete.current The number of documents currently being deleted from an index on the primary shards. |
document / None Type: float |
| elasticsearch.primaries.indexing.delete.time The total time spent deleting documents from an index on the primary shards. |
second / None Type: float |
| elasticsearch.primaries.indexing.delete.total The total number of documents deleted from an index on the primary shards. |
document / None Type: float |
| elasticsearch.primaries.indexing.index.current The number of documents currently being indexed to an index on the primary shards. |
document / None Type: float |
| elasticsearch.primaries.indexing.index.time The total time spent indexing documents to an index on the primary shards. |
second / None Type: float |
| elasticsearch.primaries.indexing.index.total The total number of documents indexed to an index on the primary shards. |
document / None Type: float |
| elasticsearch.primaries.merges.current The number of currently active segment merges on the primary shards. |
merge / None Type: float |
| elasticsearch.primaries.merges.current.docs The number of documents across segments currently being merged on the primary shards. |
document / None Type: float |
| elasticsearch.primaries.merges.current.size The size of the segments currently being merged on the primary shards. |
byte / None Type: float |
| elasticsearch.primaries.merges.total The total number of segment merges on the primary shards. |
merge / None Type: float |
| elasticsearch.primaries.merges.total.docs The total number of documents across all merged segments on the primary shards. |
document / None Type: float |
| elasticsearch.primaries.merges.total.size The total size of all merged segments on the primary shards. |
byte / None Type: float |
| elasticsearch.primaries.merges.total.time The total time spent on segment merging on the primary shards. |
second / None Type: float |
| elasticsearch.primaries.refresh.total The total number of index refreshes on the primary shards. |
refresh / None Type: float |
| elasticsearch.primaries.refresh.total.time The total time spent on index refreshes on the primary shards. |
second / None Type: float |
| elasticsearch.primaries.search.fetch.current The number of query fetches currently running on the primary shards. |
fetch / None Type: float |
| elasticsearch.primaries.search.fetch.time The total time spent on query fetches on the primary shards. |
second / None Type: float |
| elasticsearch.primaries.search.fetch.total The total number of query fetches on the primary shards. |
fetch / None Type: float |
| elasticsearch.primaries.search.query.current The number of currently active queries on the primary shards. |
query / None Type: float |
| elasticsearch.primaries.search.query.time The total time spent querying on the primary shards. |
second / None Type: float |
| elasticsearch.primaries.search.query.total The total number of queries to the primary shards. |
query / None Type: float |
| elasticsearch.primaries.store.size The total size of all the primary shards. |
byte / None Type: float |
| elasticsearch.process.open_fd |
/ Type: float |
| elasticsearch.refresh.total |
/ Type: float |
| elasticsearch.refresh.total.time |
/ Type: float |
| elasticsearch.relocating_shards |
/ Type: float |
| elasticsearch.search.active |
/ Type: float |
| elasticsearch.search.fetch.current |
/ Type: float |
| elasticsearch.search.fetch.open_contexts |
/ Type: float |
| elasticsearch.search.fetch.time |
second / Type: float |
| elasticsearch.search.fetch.total |
/ Type: float |
| elasticsearch.search.query.current |
/ Type: float |
| elasticsearch.search.query.time |
second / Type: float |
| elasticsearch.search.query.total |
/ Type: float |
| elasticsearch.search.queue |
/ Type: float |
| elasticsearch.search.threads |
/ Type: float |
| elasticsearch.store.size |
byte / Type: float |
| elasticsearch.thread_pool.bulk.active |
/ Type: float |
| elasticsearch.thread_pool.bulk.queue |
/ Type: float |
| elasticsearch.thread_pool.bulk.rejected |
/ Type: float |
| elasticsearch.thread_pool.bulk.threads |
/ Type: float |
| elasticsearch.thread_pool.fetch_shard_started.active The number of active threads in the fetch shard started pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_started.queue The number of queued threads in the fetch shard started pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_started.rejected The number of rejected threads in the fetch shard started pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_started.threads The total number of threads in the fetch shard started pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_store.active The number of active threads in the fetch shard store pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_store.queue The number of queued threads in the fetch shard store pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_store.rejected The number of rejected threads in the fetch shard store pool. |
thread / None Type: float |
| elasticsearch.thread_pool.fetch_shard_store.threads The total number of threads in the fetch shard store pool. |
thread / None Type: float |
| elasticsearch.thread_pool.flush.active |
/ Type: float |
| elasticsearch.thread_pool.flush.queue |
/ Type: float |
| elasticsearch.thread_pool.flush.rejected The number of rejected threads in the flush pool. |
thread / None Type: float |
| elasticsearch.thread_pool.flush.threads |
/ Type: float |
| elasticsearch.thread_pool.force_merge.active The number of active threads for force merge operations. |
thread / None Type: float |
| elasticsearch.thread_pool.force_merge.queue The number of queued threads for force merge operations. |
thread / None Type: float |
| elasticsearch.thread_pool.force_merge.rejected The number of rejected threads for force merge operations. |
thread / None Type: float |
| elasticsearch.thread_pool.force_merge.threads The total number of threads for force merge operations. |
thread / None Type: float |
| elasticsearch.thread_pool.generic.active |
/ Type: float |
| elasticsearch.thread_pool.generic.queue |
/ Type: float |
| elasticsearch.thread_pool.generic.rejected The number of rejected threads in the generic pool. |
thread / None Type: float |
| elasticsearch.thread_pool.generic.threads |
/ Type: float |
| elasticsearch.thread_pool.get.active |
/ Type: float |
| elasticsearch.thread_pool.get.queue |
/ Type: float |
| elasticsearch.thread_pool.get.rejected The number of rejected threads in the get pool. |
thread / None Type: float |
| elasticsearch.thread_pool.get.threads |
/ Type: float |
| elasticsearch.thread_pool.index.active |
/ Type: float |
| elasticsearch.thread_pool.index.queue |
/ Type: float |
| elasticsearch.thread_pool.index.rejected The number of rejected threads in the index pool. |
thread / None Type: float |
| elasticsearch.thread_pool.index.threads |
/ Type: float |
| elasticsearch.thread_pool.listener.active The number of active threads in the listener pool. |
thread / None Type: float |
| elasticsearch.thread_pool.listener.queue The number of queued threads in the listener pool. |
thread / None Type: float |
| elasticsearch.thread_pool.listener.rejected The number of rejected threads in the listener pool. |
thread / None Type: float |
| elasticsearch.thread_pool.listener.threads The total number of threads in the listener pool. |
thread / None Type: float |
| elasticsearch.thread_pool.management.active |
/ Type: float |
| elasticsearch.thread_pool.management.queue |
/ Type: float |
| elasticsearch.thread_pool.management.rejected The number of rejected threads in the management pool. |
thread / None Type: float |
| elasticsearch.thread_pool.management.threads |
/ Type: float |
| elasticsearch.thread_pool.merge.active The number of active threads in the merge pool. |
thread / None Type: float |
| elasticsearch.thread_pool.merge.queue The number of queued threads in the merge pool. |
thread / None Type: float |
| elasticsearch.thread_pool.merge.rejected The number of rejected threads in the merge pool. |
thread / None Type: float |
| elasticsearch.thread_pool.merge.threads The total number of threads in the merge pool. |
thread / None Type: float |
| elasticsearch.thread_pool.percolate.active |
/ Type: float |
| elasticsearch.thread_pool.percolate.queue |
/ Type: float |
| elasticsearch.thread_pool.percolate.rejected The number of rejected threads in the percolate pool. |
thread / None Type: float |
| elasticsearch.thread_pool.percolate.threads |
/ Type: float |
| elasticsearch.thread_pool.refresh.active |
/ Type: float |
| elasticsearch.thread_pool.refresh.queue |
/ Type: float |
| elasticsearch.thread_pool.refresh.rejected The number of rejected threads in the refresh pool. |
thread / None Type: float |
| elasticsearch.thread_pool.refresh.threads |
/ Type: float |
| elasticsearch.thread_pool.search.active |
/ Type: float |
| elasticsearch.thread_pool.search.queue |
/ Type: float |
| elasticsearch.thread_pool.search.rejected The number of rejected threads in the search pool. |
thread / None Type: float |
| elasticsearch.thread_pool.search.threads |
/ Type: float |
| elasticsearch.thread_pool.snapshot.active |
/ Type: float |
| elasticsearch.thread_pool.snapshot.queue |
/ Type: float |
| elasticsearch.thread_pool.snapshot.rejected The number of rejected threads in the snapshot pool. |
thread / None Type: float |
| elasticsearch.thread_pool.snapshot.threads |
/ Type: float |
| elasticsearch.transport.rx_count |
/ Type: float |
| elasticsearch.transport.rx_size |
/ Type: float |
| elasticsearch.transport.server_open |
/ Type: float |
| elasticsearch.transport.tx_count |
/ Type: float |
| elasticsearch.transport.tx_size |
byte / Type: float |
| elasticsearch.unassigned_shards |
/ Type: float |
| jvm.gc.collection_count The total number of garbage collections run by the JVM. |
garbage collection / None Type: float |
| jvm.gc.collection_time The total time spent on garbage collection in the JVM. |
second / None Type: float |
| jvm.gc.collectors.old.collection_time The total time spent in major GCs in the JVM that collect old generation objects. |
second / None Type: float |
| jvm.gc.collectors.old.count The total count of major GCs in the JVM that collect old generation objects. |
garbage collection / None Type: float |
| jvm.gc.collectors.young.collection_time The total time spent in minor GCs in the JVM that collects young generation objects. |
second / None Type: float |
| jvm.gc.collectors.young.count The total count of minor GCs in the JVM that collects young generation objects. |
garbage collection / None Type: float |
| jvm.gc.concurrent_mark_sweep.collection_time The total time spent on "concurrent mark & sweep" GCs in the JVM. |
second / None Type: float |
| jvm.gc.concurrent_mark_sweep.count The total count of "concurrent mark & sweep" GCs in the JVM. |
garbage collection / None Type: float |
| jvm.gc.par_new.collection_time The total time spent on "parallel new" GCs in the JVM. |
second / None Type: float |
| jvm.gc.par_new.count The total count of "parallel new" GCs in the JVM. |
garbage collection / None Type: float |
| jvm.mem.heap_committed The amount of memory guaranteed to be available to the JVM heap. |
byte / None Type: float |
| jvm.mem.heap_in_use The amount of memory currently used by the JVM heap as a value between 0 and 1. |
None / None Type: float |
| jvm.mem.heap_max The maximum amount of memory that can be used by the JVM heap. |
byte / None Type: float |
| jvm.mem.heap_used The amount of memory in bytes currently used by the JVM heap. |
byte / None Type: float |
| jvm.mem.non_heap_committed The amount of memory guaranteed to be available to JVM non-heap. |
byte / None Type: float |
| jvm.mem.non_heap_used The amount of memory in bytes currently used by the JVM non-heap. |
byte / None Type: float |
| jvm.threads.count The number of active threads in the JVM. |
thread / None Type: float |
| jvm.threads.peak_count The peak number of threads used by the JVM. |
thread / None Type: float |
Comments