Skip to content
Bleemeo Bleemeo

Kafka

Auto-Discovery
Health Check
Metrics
Logs

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

In addition, Kafka needs to expose JMX over a TCP port.

If you are using Docker, add the environment variable KAFKA_JMX_PORT=1234, see the documentation for details.

In other cases, you need to export some environment variable before running kafka-server-start.sh:

Terminal window
export JMX_PORT=1234
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1"

Warning: this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

The Bleemeo agent should auto-detect the JMX port, if some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]


  # For a Kafka running outside a container
  - type: "kafka"
    address: "127.0.0.1"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"


  # For an additional Kafka running outside a container
  - type: "kafka"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"


  # For a Kafka running in a Docker container
  - type: "kafka"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

The following metrics are gathered:

Metric Description
service_status Status of Kafka
kafka_jvm_gc Number of garbage collection per second
kafka_jvm_gc_utilization Garbage collection utilization in percent
kafka_jvm_heap_used Heap memory used in bytes
kafka_jvm_non_heap_used Non-Heap memory used in bytes
kafka_topics_count Total number of topics
kafka_fetch_requests_sum Total number of fetch requests per second
kafka_fetch_time_average Average time to process a fetch request in seconds
kafka_produce_requests_sum Total number of produce requests per second
kafka_produce_time_average Average time to process a produce request in seconds

Bleemeo also supports detailed monitoring of specific Kafka topics. To enable this, add the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]


  # For a Kafka running outside a container
  - type: "kafka"
    detailed_items:
      - "topic1"
      - "topic2"
    [...]


  # For a Kafka running in a Docker container
  - type: "kafka"
    instance: "CONTAINER_NAME"
    detailed_items:
      - "topic1"
      - "topic2"
    [...]

When using Docker, you may use labels to set the detailed topics:

Terminal window
docker run --label glouton.detailed_items="topic1,topic2" [...]

The following per-topic metrics will be gathered:

Metric Description
kafka_fetch_requests Number of fetch requests per second
kafka_produce_requests Number of produce requests per second

If you want to enable more JMX metrics, you can add custom JMX metrics. A list of all the available metrics is available here.