Monitoring Kubernetes

This article will help you get the Kubernetes plugin for sd-agent configured and returning metrics

Installing the kubernetes plugin package

Install the kubernetes plugin on Debian/Ubuntu:

sudo apt-get install sd-agent-kubernetes

Install the kubernetes plugin on RHEL/CentOS:

sudo yum install sd-agent-kubernetes

Read more about agent plugins.

Configuring the agent to monitor Kubernetes DNS

1. Configure /etc/sd-agent/conf.d/kubernetes.yaml

init_config:
  # Initialization retries
  #
  # if the agent is expected to start before Kubelet,
  # use these settings to configure the retry policy.
  #
  # init_retry_interval defines how long (in seconds) the kubelet client
  # will wait before retrying initialization.
  # Defaults to 0.
  #
  # init_retry_interval: 20
  #
  # init_retries configures how many retries are made before failing permanently.
  # Defaults to 0.
  #
  # init_retries: 5

instances:
  # The kubernetes check retrieves metrics from cadvisor running under kubelet on each node.
  # It also queries kubelet for its health and the list of local-running pods, and (optionally)
  # the apiserver for services and events
  #
  # By default we assume we're running under docker and
  # that the kubelet read-only port with no auth is enabled.
  #
  # In this case we will use the address of the default router to reach the kubelet and cadvisor APIs
  # unless the environment variable KUBERNETES_KUBELET_HOST is found. You can set this variable to
  # spec.nodeName (If your node name can be resolved by pods) or status.hostIP (for Kubernetes 1.7+)
  # through the downward API. See
  # https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
  #
  # To enable Kubernetes State Metrics, please refer to kubernetes_states integration documentation.
  #
  # If the read-only endpoint is disabled, the check will query kubelet over HTTPS
  #
  # To override this behavior, e.g. in the case of a standalone cadvisor instance, use the following:
  #
  # host: localhost
  # port: 4194
  # method: http

  # cAdvisor port, set it to 0 if cAdvisor is unavailable
 - port: 4194
  #
  # cAdvisor host
  # host: localhost
  #
  # kubelet port. It needs to be set if you are not using a default one (10250 or 10255)
  # kubelet_port: 10255
  #
  # apiserver url for cluster-level queries. If not configured here, the KUBERNETES_SERVICE_HOST
  # and KUBERNETES_SERVICE_PORT environment variables will be used
  #
  # api_server_url: https://kubernetes:443

  # Client Authentication against the apiserver and kubelet
  #
  # By default the agent authenticates against the apiserver and kubelet with its service account
  # bearer token. If you want to specify its path, set the following option. If X509 client certificates
  # are set, either for the kubelet or apiserver, they will be used instead. The recommended way to expose
  # these files to the agent is by using Kubernetes Secrets.
  #
  # bearer_token_path: /var/run/secrets/kubernetes.io/serviceaccount/token
  #
  # apiserver_client_crt: /path/to/client.crt
  # apiserver_client_key: /path/to/client.key
  #
  # kubelet_client_crt: /path/to/client.crt
  # kubelet_client_key: /path/to/client.key

  # Server Authentication for apiserver and kubelet
  #
  # Similarly we use the default CA cert of the agent's service account to verify the
  # apiserver's identity, but custom ones can be specified here.
  #
  # apiserver_ca_cert: /path/to/cacert.crt
  # kubelet_cert: /path/to/ca.pem
  #
  # The default for kubelet traffic is to try and use the read-only port that doesn't require TLS
  # and to fall back to the HTTPS API with simple TLS validation. Providing a cert forces TLS validation on.
  # Explicitly disabling tls_verify should be used with caution:
  # if an attacker sniffs the agent requests they will see the agent's service account bearer token.
  #
  # kubelet_tls_verify: True

Ensure that the options correctly point to your server and port.

2. Restart the agent

sudo /etc/init.d/sd-agent restart

or

sudo systemctl restart sd-agent

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 

or

/usr/share/python/sd-agent/agent.py info

If the agent has been configured correctly you'll see an output such as:

kubernetes
-----
  - instance #0 [OK]
  - Collected * metrics

You can also view the metrics returned with the following command:

sudo -u sd-agent /usr/share/python/sd-agent/agent.py check kubernetes

Configuring graphs

Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the kubernetes metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

Screen_Shot_2018-01-18_at_11.46.12.png

Monitored metrics

MetricValues
kubernetes.cpu.capacity

The number of cores in this machine.
None / None
Type: float
kubernetes.cpu.limits

The limit of cpu cores set
cpu / None
Type: float
kubernetes.cpu.requests

The requested cpu cores
cpu / None
Type: float
kubernetes.cpu.usage.total

The percentage of CPU time used
percent_nano / None
Type: float
kubernetes.filesystem.usage

The amount of disk used
byte / None
Type: float
kubernetes.filesystem.usage_pct

The percentage of disk used
fraction / None
Type: float
kubernetes.memory.capacity

The amount of memory (in bytes) in this machine
byte / None
Type: float
kubernetes.memory.limits

The limit of memory set
byte / None
Type: float
kubernetes.memory.requests

The requested memory
byte / None
Type: float
kubernetes.memory.usage

The amount of memory used
byte / None
Type: float
kubernetes.network.rx_bytes

The amount of bytes per second received
byte / second
Type: float
kubernetes.network.tx_bytes

The amount of bytes per second transmitted
byte / second
Type: float
kubernetes.network_errors

The amount of network errors per second
error / second
Type: float
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Monday  —  Friday.

10am  —  6pm UK.

Dedicated Support.