Monitoring Kubernetes

This article will help you get the Kubernetes plugin for sd-agent configured and returning metrics

Installing the kubernetes plugin package

Install the kubernetes plugin on Debian/Ubuntu:

sudo apt-get install sd-agent-kubernetes

Install the kubernetes plugin on RHEL/CentOS:

sudo yum install sd-agent-kubernetes

Read more about agent plugins.

Configuring the agent to monitor Kubernetes DNS

1. Configure /etc/sd-agent/conf.d/kubernetes.yaml

init_config:
  # Initialization retries
  #
  # if the agent is expected to start before Kubelet,
  # use these settings to configure the retry policy.
  #
  # init_retry_interval defines how long (in seconds) the kubelet client
  # will wait before retrying initialization.
  # Defaults to 0.
  #
  # init_retry_interval: 20
  #
  # init_retries configures how many retries are made before failing permanently.
  # Defaults to 0.
  #
  # init_retries: 5

instances:
  # The kubernetes check retrieves metrics from cadvisor running under kubelet on each node.
  # It also queries kubelet for its health and the list of local-running pods, and (optionally)
  # the apiserver for services and events
  #
  # By default we assume we're running under docker and
  # that the kubelet read-only port with no auth is enabled.
  #
  # In this case we will use the address of the default router to reach the kubelet and cadvisor APIs
  # unless the environment variable KUBERNETES_KUBELET_HOST is found. You can set this variable to
  # spec.nodeName (If your node name can be resolved by pods) or status.hostIP (for Kubernetes 1.7+)
  # through the downward API. See
  # https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
  #
  # To enable Kubernetes State Metrics, please refer to kubernetes_states integration documentation.
  #
  # If the read-only endpoint is disabled, the check will query kubelet over HTTPS
  #
  # To override this behavior, e.g. in the case of a standalone cadvisor instance, use the following:
  #
  # host: localhost
  # port: 4194
  # method: http

  # cAdvisor port, set it to 0 if cAdvisor is unavailable
 - port: 4194
  #
  # cAdvisor host
  # host: localhost
  #
  # kubelet port. It needs to be set if you are not using a default one (10250 or 10255)
  # kubelet_port: 10255
  #
  # apiserver url for cluster-level queries. If not configured here, the KUBERNETES_SERVICE_HOST
  # and KUBERNETES_SERVICE_PORT environment variables will be used
  #
  # api_server_url: https://kubernetes:443

  # Client Authentication against the apiserver and kubelet
  #
  # By default the agent authenticates against the apiserver and kubelet with its service account
  # bearer token. If you want to specify its path, set the following option. If X509 client certificates
  # are set, either for the kubelet or apiserver, they will be used instead. The recommended way to expose
  # these files to the agent is by using Kubernetes Secrets.
  #
  # bearer_token_path: /var/run/secrets/kubernetes.io/serviceaccount/token
  #
  # apiserver_client_crt: /path/to/client.crt
  # apiserver_client_key: /path/to/client.key
  #
  # kubelet_client_crt: /path/to/client.crt
  # kubelet_client_key: /path/to/client.key

  # Server Authentication for apiserver and kubelet
  #
  # Similarly we use the default CA cert of the agent's service account to verify the
  # apiserver's identity, but custom ones can be specified here.
  #
  # apiserver_ca_cert: /path/to/cacert.crt
  # kubelet_cert: /path/to/ca.pem
  #
  # The default for kubelet traffic is to try and use the read-only port that doesn't require TLS
  # and to fall back to the HTTPS API with simple TLS validation. Providing a cert forces TLS validation on.
  # Explicitly disabling tls_verify should be used with caution:
  # if an attacker sniffs the agent requests they will see the agent's service account bearer token.
  #
  # kubelet_tls_verify: True

Ensure that the options correctly point to your server and port.

2. Restart the agent

sudo /etc/init.d/sd-agent restart

or

sudo systemctl restart sd-agent

Verifying the configuration
Execute info to verify the configuration with the following:

sudo /etc/init.d/sd-agent info 

or

/usr/share/python/sd-agent/agent.py info

If the agent has been configured correctly you'll see an output such as:

kubernetes
-----
  - instance #0 [OK]
  - Collected * metrics

You can also view the metrics returned with the following command:

sudo -u sd-agent /usr/share/python/sd-agent/agent.py check kubernetes

Configuring graphs

Click the name of your server from the Devices list in your Server Density account then go to the Metrics tab. Click the + Graph button on the right then choose the kubernetes metrics to display the graphs. The metrics will also be available to select when building dashboard graphs.

Monitored metrics

Metric	Values
kubernetes.cpu.capacity The number of cores in this machine.	None / None Type: float
kubernetes.cpu.limits The limit of cpu cores set	cpu / None Type: float
kubernetes.cpu.requests The requested cpu cores	cpu / None Type: float
kubernetes.cpu.usage.total The percentage of CPU time used	percent_nano / None Type: float
kubernetes.filesystem.usage The amount of disk used	byte / None Type: float
kubernetes.filesystem.usage_pct The percentage of disk used	fraction / None Type: float
kubernetes.memory.capacity The amount of memory (in bytes) in this machine	byte / None Type: float
kubernetes.memory.limits The limit of memory set	byte / None Type: float
kubernetes.memory.requests The requested memory	byte / None Type: float
kubernetes.memory.usage The amount of memory used	byte / None Type: float
kubernetes.network.rx_bytes The amount of bytes per second received	byte / second Type: float
kubernetes.network.tx_bytes The amount of bytes per second transmitted	byte / second Type: float
kubernetes.network_errors The amount of network errors per second	error / second Type: float