Skip to content
Bleemeo Bleemeo

Kafka

Auto-Detection
Built-in Metrics
Logs

Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications.

Auto-Detection

Section titled “Auto-Detection”

Glouton detects Kafka by its process and listening port (default: 9092).

To enable metrics gathering, Glouton needs JMX access. See Java Metrics for setup details.

Kafka must expose JMX over a TCP port. If you are using Docker, add the environment variable KAFKA_JMX_PORT=1234 — see the Confluent documentation for details.

In other cases, export the following environment variables before running kafka-server-start.sh:

Terminal window
export JMX_PORT=1234
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1"

Glouton auto-detects the JMX port. If auto-detected parameters are incorrect, override them manually:

Terminal window
sudo tee /etc/glouton/conf.d/99-kafka.conf > /dev/null << 'EOF'
service:
  # For a Kafka running outside a container
  - type: "kafka"
    address: "127.0.0.1"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole"   # by default, no authentication is done
    jmx_password: "secret"


  # For an additional Kafka running outside a container
  - type: "kafka"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole"
    jmx_password: "secret"


  # For a Kafka running in a Docker container
  - type: "kafka"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole"
    jmx_password: "secret"
EOF

Glouton automatically detects configuration changes.

Built-in Metrics

Section titled “Built-in Metrics”
Metric Description
service_status Status of Kafka
kafka_jvm_gc Number of garbage collections per second
kafka_jvm_gc_utilization Garbage collection utilization in percent
kafka_jvm_heap_used Heap memory used in bytes
kafka_jvm_non_heap_used Non-heap memory used in bytes
kafka_topics_count Total number of topics
kafka_fetch_requests_sum Total number of fetch requests per second
kafka_fetch_time_average Average time to process a fetch request in seconds
kafka_produce_requests_sum Total number of produce requests per second
kafka_produce_time_average Average time to process a produce request in seconds

Bleemeo also supports detailed monitoring of specific Kafka topics. To enable this, add detailed_items to the service configuration:

Terminal window
sudo tee /etc/glouton/conf.d/99-kafka-topics.conf > /dev/null << 'EOF'
service:
  - type: "kafka"
    detailed_items:
      - "topic1"
      - "topic2"
EOF

When using Docker, you can set detailed topics via labels:

Terminal window
docker run --label glouton.detailed_items="topic1,topic2" [...]

The following per-topic metrics are gathered:

Metric Description
kafka_fetch_requests Number of fetch requests per second
kafka_produce_requests Number of produce requests per second

To enable additional JMX metrics, you can add custom JMX metrics. A full list of available metrics is available in the Kafka documentation.

Logs

Section titled “Logs”

Glouton collects Kafka logs for analysis in Bleemeo.