Skip to main content

Kubernetes Metrics

Professional

Bleemeo agent support discovering services running on Kubernetes. It will monitor all known services and gather Docker metrics.

But to achieve this, it require access to Kubernetes API. The easiest is to run Bleemeo agent in the Kubernetes cluster using DaemonSet. You should use the deployment yaml provided, you only need to change GLOUTON_BLEEMEO_ACCOUNT_ID, GLOUTON_BLEEMEO_REGISTRATION_KEY and GLOUTON_KUBERNETES_CLUSTERNAME.

The following use envsubst (should be available on all Linux) to automatically replace the values:

export GLOUTON_BLEEMEO_ACCOUNT_ID=<YOUR-ACCOUNT-ID>
export GLOUTON_BLEEMEO_REGISTRATION_KEY=<YOUR-REGISTRATION-KEY>
export GLOUTON_KUBERNETES_CLUSTERNAME="my_k8s_cluster_name"
curl -s https://packages.bleemeo.com/bleemeo-agent/kubernetes/k8s.yaml | envsubst | kubectl apply -f -

This will start one Bleemeo agent on every node of your Kubernetes cluster.

In addition to metrics from known services, Glouton will monitor:

  • kubernetes_pods_count: The number of pods in a specific state, with the following labels:
    • owner_kind: the kind of the pod's owner, e.g. daemonset, deployment...
    • owner_name: the name of the pod's owner, e.g. glouton, kube-proxy.
    • state: the current state of the pod (pending, running, succeeded or failed).
    • namespace: the pod's namespace.
  • kubernetes_cpu_requests, kubernetes_cpu_limits: The sum of cpu requests and limits inside a Kubernetes workload, with the following labels:
    • owner_kind: the kind of the workload, e.g. daemonset, deployment...
    • owner_name: the name of the workload, e.g. glouton, kube-proxy.
    • namespace: the workload's namespace.
  • kubernetes_memory_requests, kubernetes_memory_limits: The sum of memory requests and limits inside a Kubernetes workload, in bytes, with the following labels:
    • owner_kind: the kind of the workload, e.g. daemonset, deployment...
    • owner_name: the name of the workload, e.g. glouton, kube-proxy.
    • namespace: the workload's namespace.
  • kubernetes_pods_restart_count: The number of restarts of a pod, with the following labels:
    • owner_kind: the kind of the pod's owner, e.g. daemonset, deployment...
    • owner_name: the name of the pod's owner, e.g. glouton, kube-proxy.
    • pod_name: the pod's name.
    • namespace: the pod's namespace.
  • kubernetes_namespaces_count: The number of namespaces, with the state label (it can either be active or terminating).
  • kubernetes_nodes_count: The number of nodes in the cluster.
  • kubernetes_ca_day_left: Time left before Kubernetes CA expiry in days.
  • kubernetes_certificate_day_left: Time left before Kubernetes certificate expiry in days.
  • kubernetes_api_status: Status of Kubernetes API server running on same node as the Glouton.
  • kubernetes_kubelet_status: Status of the Kubelet running on the same node as the Glouton. This status is based on what the Kubernetes API knows of the Kubelet node, it covers low resources available (disk, memory and processes) and unready node.

Finally all Bleemeo agent on a same cluster will together monitor the status of the Kubernetes API, and report an error if the majority fail to contact the Kubernetes API. This status will be visible in the Status Dashboard and will use the cluster name your provided (GLOUTON_KUBERNETES_CLUSTERNAME used during installation).

Kubernetes annotations​

The following Pod annotations are known by the Bleemeo agent:

  • glouton.enable: If false, the Bleemeo agent will ignore the Pod. Once ignored, information and metrics from this Pod won't be gathered and any service running in this Pod won't be discovered.
  • glouton.check.ignore.port.80: If true, the Bleemeo agent skips the check on port 80 that the auto-discovered may have wrongly detected. This option isn't limited to port 80, and any other value could be used.
  • prometheus.io/scrape, prometheus.io/path, prometheus.io/port: Configure the Bleemeo agent to scrape a Prometheus metric exporter. See See the Prometheus page for details.
  • glouton.allow_metrics, glouton.deny_metrics, glouton.include_default_metrics: Configure which Prometheus metrics are scraped. See See the Prometheus page for details.

Note that the annotations must be on the Pod itself. For example on a Kubernetes deployment, the yaml will look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "my-application"
  # annotations here would be annotation on the Deployment, not on the Pod
spec:
  template:
    metadata:
      annotations:
        glouton.check.ignore.port.80: "true"
    spec:
      containers:
      - name: "my-application"
        image: "hello-world"

Custom Configuration​

If you need to some custom configuration to your Bleemeo agent, you will need to change the Kubernetes config map "glouton-config". We recommend to not change directly the sample k8s.yaml, to allow to easier update. Instead we recommend to:

  • Create another YAML file that only define the config map "glouton-config"
  • Apply both the sample k8s.yaml AND your custom file for each update

For example, create a custom file "glouton-custom.yaml" to allow some custom metrics (see Filtering page for details):

cat > glouton-custom.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: glouton-config
data:
  99-local.conf: |
    # Add Glouton customization here
    metric:
      allow_metrics:
        - "django_http_responses_total_by_status_total"
        - "django_http_requests_latency_seconds_by_view_method_sum"
        - "django_http_requests_latency_seconds_by_view_method_count"
EOF

Then to apply your change:

export GLOUTON_BLEEMEO_ACCOUNT_ID=<YOUR-ACCOUNT-ID>
export GLOUTON_BLEEMEO_REGISTRATION_KEY=<YOUR-REGISTRATION-KEY>
export GLOUTON_KUBERNETES_CLUSTERNAME="my_k8s_cluster_name"
curl -s https://packages.bleemeo.com/bleemeo-agent/kubernetes/k8s.yaml | envsubst | kubectl apply -f -,./glouton-custom.yaml