Kubernetes Metrics

Professional

The Bleemeo agent supports discovering services running on Kubernetes. It will monitor all known services and gather Docker metrics.

To do this, it requires access to the Kubernetes API. The easiest way is to run the Bleemeo agent in the Kubernetes cluster using a DaemonSet. You should use the deployment yaml provided, you only need to change GLOUTON_BLEEMEO_ACCOUNT_ID, GLOUTON_BLEEMEO_REGISTRATION_KEY and GLOUTON_KUBERNETES_CLUSTERNAME.

The following uses envsubst (it should be available on all Linux distributions) to automatically replace the values:

export GLOUTON_BLEEMEO_ACCOUNT_ID=<YOUR-ACCOUNT-ID>
export GLOUTON_BLEEMEO_REGISTRATION_KEY=<YOUR-REGISTRATION-KEY>
export GLOUTON_KUBERNETES_CLUSTERNAME="my_k8s_cluster_name"
curl -s https://packages.bleemeo.com/bleemeo-agent/kubernetes/k8s.yaml | envsubst | kubectl apply -f -

This will start one Bleemeo agent on every node of your Kubernetes cluster.

In addition to metrics from known services, Glouton will monitor:

kubernetes_pods_count: The number of pods in a specific state, with the following labels:
- owner_kind: the kind of the pod’s owner, e.g. daemonset, deployment…
- owner_name: the name of the pod’s owner, e.g. glouton, kube-proxy.
- state: the current state of the pod (pending, running, succeeded or failed).
- namespace: the pod’s namespace.
kubernetes_cpu_requests, kubernetes_cpu_limits: The sum of CPU requests and limits inside a Kubernetes workload, with the following labels:
- owner_kind: the kind of the workload, e.g. daemonset, deployment…
- owner_name: the name of the workload, e.g. glouton, kube-proxy.
- namespace: the workload’s namespace.
kubernetes_memory_requests, kubernetes_memory_limits: The sum of memory requests and limits inside a Kubernetes workload, in bytes, with the following labels:
- owner_kind: the kind of the workload, e.g. daemonset, deployment…
- owner_name: the name of the workload, e.g. glouton, kube-proxy.
- namespace: the workload’s namespace.
kubernetes_pods_restart_count: The number of restarts of a pod, with the following labels:
- owner_kind: the kind of the pod’s owner, e.g. daemonset, deployment…
- owner_name: the name of the pod’s owner, e.g. glouton, kube-proxy.
- pod_name: the pod’s name.
- namespace: the pod’s namespace.
kubernetes_replicas_desired, kubernetes_replicas_ready, kubernetes_replicas_available: The desired number of replicas of a Kubernetes workload, how many of them are ready, and how many are available (ready long enough to satisfy the workload’s minReadySeconds). available is reported only for Deployments, StatefulSets and DaemonSets. Labels:
- owner_kind: the kind of the workload, e.g. deployment, statefulset, daemonset…
- owner_name: the name of the workload, e.g. glouton, kube-proxy.
- namespace: the workload’s namespace.
kubernetes_hpa_min_replicas, kubernetes_hpa_max_replicas: The minimum and maximum replica bounds configured on a HorizontalPodAutoscaler. Comparing kubernetes_replicas_desired with the maximum lets you detect an autoscaler pinned at its ceiling. The metrics are labeled by the HPA’s scale target, so they join naturally with the kubernetes_replicas_* metrics:
- owner_kind: the kind of the scale target, e.g. deployment, statefulset.
- owner_name: the name of the scale target.
- namespace: the HorizontalPodAutoscaler’s namespace.
kubernetes_hpa_scaling_limited: 1 when the HorizontalPodAutoscaler wants to scale beyond its min/max bounds but is clamped to them, 0 otherwise. Same labels as above.
kubernetes_hpa_status: A self-declared health status of the HorizontalPodAutoscaler (it does not use user-defined thresholds): critical when the HPA cannot read its metrics or act on its target, warning when scaling is intentionally disabled (target scaled to 0 replicas), ok otherwise. Same labels as above.
kubernetes_namespaces_count: The number of namespaces, with the state label (it can either be active or terminating).
kubernetes_nodes_count: The number of nodes in the cluster.
kubernetes_cpu_allocatable, kubernetes_memory_allocatable, kubernetes_pods_allocatable, kubernetes_ephemeral_storage_allocatable: The allocatable CPU (in cores), memory (in bytes), number of pods and ephemeral storage (in bytes) of the node running Glouton.
kubernetes_ca_day_left: Time left before Kubernetes CA expiry in days.
kubernetes_certificate_day_left: Time left before Kubernetes certificate expiry in days.
kubernetes_api_status: Status of the Kubernetes API server running on the same node as Glouton.
kubernetes_kubelet_status: Status of the Kubelet running on the same node as the Glouton. This status is based on what the Kubernetes API knows of the Kubelet node, it covers low resources available (disk, memory and processes) and unready node. One metric is reported per node condition, identified by the condition label (ready, disk_pressure, memory_pressure, pid_pressure or network_unavailable).
kubernetes_job_failed_pods: The number of failed pod attempts of the latest Job, with the following labels:
- owner_kind: the kind of the owner, either job (standalone Job) or cronjob.
- owner_name: the name of the Job or CronJob.
- namespace: the namespace.
kubernetes_last_job_duration_seconds: The execution duration, in seconds, of the most recent completed run, with the same owner_kind, owner_name and namespace labels as above.
kubernetes_cronjob_missed_runs: The number of scheduled CronJob runs that should have succeeded by now but did not, with owner_kind (cronjob), owner_name and namespace labels.
kubernetes_cronjob_last_success_age_seconds: The time, in seconds, since the CronJob’s last successful run. It is not reported if the CronJob never succeeded. Same labels as above.

Finally, all Bleemeo agents on the same cluster will together monitor the status of the Kubernetes API and report an error if the majority fail to contact it. This status will be visible in the Status Dashboard and will use the cluster name you provided (GLOUTON_KUBERNETES_CLUSTERNAME used during installation).

Persistent Volume metrics

When a pod mounts a PersistentVolumeClaim, Glouton attaches Kubernetes labels to the filesystem (disk_*) and block device (io_*) metrics of that volume, so you can attribute storage usage and I/O to the workload that owns them:

namespace: the pod’s namespace.
pod_name: the pod’s name.
owner_kind, owner_name: the kind and name of the pod’s owner.
pv: the PersistentVolume name, for a PVC-backed volume.
volume: the in-pod volume name (spec.volumes[].name).

The item label of these metrics is also rewritten to a stable value — the PersistentVolume name (PVC-backed volume) or pod_name/volume (inline volume) — instead of the volatile kubelet mount path.

Kubernetes annotations

The following Pod annotations are known by the Bleemeo agent:

glouton.enable: If false, the Bleemeo agent will ignore the Pod. Once ignored, information and metrics from this Pod won’t be gathered and any service running in this Pod won’t be discovered.
glouton.check.ignore.port.80: If true, the Bleemeo agent skips the check on port 80 that auto-discovery may have wrongly detected. This option isn’t limited to port 80, and any other value could be used.
prometheus.io/scrape, prometheus.io/path, prometheus.io/port: Configure the Bleemeo agent to scrape a Prometheus metric exporter. See the Prometheus page for details.
glouton.allow_metrics, glouton.deny_metrics, glouton.include_default_metrics: Configure which Prometheus metrics are scraped. See the Prometheus page for details.

A Pod can also override a service’s configuration through glouton.<setting> annotations (for example glouton.username). For security reasons, only an allow-list of settings can be set this way; sensitive settings are restricted by default. See the service settings table for the available settings and which ones are allowed by default.

Note that the annotations must be on the Pod itself. For example on a Kubernetes deployment, the yaml will look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "my-application"
  # annotations here would be annotation on the Deployment, not on the Pod
spec:
  template:
    metadata:
      annotations:
        glouton.check.ignore.port.80: "true"
    spec:
      containers:
      - name: "my-application"
        image: "hello-world"

Custom Configuration

If you need to apply some custom configuration to your Bleemeo agent, you will need to change the Kubernetes config map “glouton-config”. We recommend not editing the sample k8s.yaml directly, so future updates stay easy to apply. Instead, we recommend:

Creating another YAML file that only defines the config map “glouton-config”
Applying both the sample k8s.yaml AND your custom file for each update

For example, create a custom file “glouton-custom.yaml” to allow some custom metrics (see Filtering page for details):

cat > glouton-custom.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: glouton-config
data:
  99-local.conf: |
    # Add Glouton customization here
    metric:
      allow_metrics:
        - "django_http_responses_total_by_status_total"
        - "django_http_requests_latency_seconds_by_view_method_sum"
        - "django_http_requests_latency_seconds_by_view_method_count"
EOF

Then to apply your change:

export GLOUTON_BLEEMEO_ACCOUNT_ID=<YOUR-ACCOUNT-ID>
export GLOUTON_BLEEMEO_REGISTRATION_KEY=<YOUR-REGISTRATION-KEY>
export GLOUTON_KUBERNETES_CLUSTERNAME="my_k8s_cluster_name"
curl -s https://packages.bleemeo.com/bleemeo-agent/kubernetes/k8s.yaml | envsubst | kubectl apply -f -,./glouton-custom.yaml