Prometheus Monitoring

Bleemeo agent could query a Prometheus exporter for custom metrics.

To configure Prometheus metrics, you just need to add the following to your Bleemeo agent configuration (/etc/glouton/conf.d/50-prometheus-metrics.conf):

metric:
  prometheus:
    targets:
      - url: "http://localhost:8080/metrics"
        name: "my_application"
        allow_metrics:
          - "custom_metric_name"
          - "my_application_metric_*"
      - url: "file:///var/run/custom_file.txt"
        name: "my_cron_metrics"
        allow_metrics:
          - "custom_metric_name"
      - url: "http://localhost:8088/metrics"
        name: "another_application"
        allow_metrics:
          - "other_metric_name_*"
        deny_metrics:
          - "other_metric_name_not_important"

Restart your agent to apply the new configuration.

As you can see, Bleemeo agent support fetching metrics from an HTTP server but also from a file. Reading from a file is useful if your application don't have an HTTP server or if you generate your metrics from a cron. The content of the file is the Prometheus text format, which is the same as HTTP exporter.

Remember that you need to allow the custom metrics you want using allow_metrics. See Filtering page for more details.

If your application is running in Docker, you can use Docker labels to configure Bleemeo agent to gather your custom metrics. The following labels are known:

• prometheus.io/scrape: if true, Bleemeo agent will gather metrics from this container. You probably need to set prometheus.io/port
• prometheus.io/port: configure the port where your application export its Prometheus metrics. The default is 9201.
• prometheus.io/path: configure the path where your application export its Prometheus metrics. The default is "/metrics".
• glouton.allow_metrics: set the allowed list of metrics. It's a coma separated list. The default is the global allow-list from configuration file.
• glouton.deny_metrics: set the denied list of metrics. It's a coma separated list. The default is the global deny-list from configuration file.

For example, running a container with the following labels:

docker run --label prometheus.io/scrape=true --label prometheus.io/port=8080 \
    --label "glouton.allow_metrics=business_counter,application_prefix_*" \
    --label "glouton.deny_metrics=application_prefix_ignore_me" \
    example/my_application:latest

Will configure Bleemeo agent to gather metrics on http://$CONTAINER_IP:8080/metrics, and collect business_counter and all metrics starting with application_prefix_ except application_prefix_ignore_me which is denied.

If your application is running in Kubernetes, you can use the Docker labels described above as Kubernetes Pod annotation. The same Docker example would translate to the following Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "my-application"
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        glouton.allow_metrics: "business_counter,application_prefix_*"
        glouton.deny_metrics: "application_prefix_ignore_me"
    spec:
      containers:
      - name: "my-application"
        image: "example/my_application:latest"

caution

The number of custom metrics is limited, so you should only send metrics that are important to you.