Services Monitoring

Bleemeo agent can discover services running on your system and automatically monitor specific metrics for such services. For example with Apache HTTP server, the number of requests served is automatically monitored. For each service detected a tag with the service name is created which allows to filter your agents by service running on them.

If you have any service not listed on this page, you can define a custom check or define a custom metric.

If you want to disable metrics for a service, you can ignore some services.

Common Features

Service Status

The agent checks TCP sockets for each service. By default, a simple TCP connection is used to test the service, but some services support a specific check. See the service details below for the supported specific checks. Those checks are executed every minute. They may be run earlier for TCP services: the Bleemeo agent keeps a connection open with the service and if that connection is broken the check is executed immediately.

The current status of the service can be viewed in the Status Dashboard and you can configure a notification to be alerted when the status changes.

The history of the status is also stored in a metric named service_status. One metric per service is created, the metric has two labels to identify the service:

service: the kind of the service, like "apache", "nginx"...
service_instance: the container name. This label is absent when the service isn't running in a container.

The value of this metric is:

0 when the check passed successfully
1 when the check passed with a warning. For example an Apache server responded with a 404 page
2 when the check detected an issue with the service
3 when the check doesn't know the status of the service. This happens when the check itself failed, usually due to a timeout

Ignore ports

By default, the agent detects and monitors all ports used by a service. You can ignore a port with the ignore_ports setting. For example, to ignore the port 80 on Apache, use the following config:

- id: "apache"
  ignore_ports:
    - 80

Docker labels

You can use docker labels to override the service config. Any setting that you use on a service can be replaced by a label using the format glouton.setting.

For instance, to ignore ports 8000 and 9000 on a container, use:

docker run --labels glouton.ignore_ports="8000,9000" [...]

Apache HTTP

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a HTTP check if the service listen on port 80.

To enable metrics gathering, ensure Bleemeo agent could access Apache status on the URL http://server-address/server-status?auto. This should be done by default with Apache on Ubuntu or Debian. If not, this usually means to add the following to your default virtual host (e.g. the first file in /etc/apache2/sites-enabled/):

LoadModule status_module /usr/lib/apache2/modules/mod_status.so

<Location /server-status>
    SetHandler server-status
    # You may want to limit access only to Bleemeo agent
    # require ip IP-OF-BLEEMEO-AGENT/32
</Location>
ExtendedStatus On

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Apache running outside any container
  - id: "apache"
    address: "127.0.0.1"
    port: 80   # HTTP listener, Agent don't support HTTPS here
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

  # For an additional Apache running outside any container
  - id: "apache"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 81   # HTTP listener, Agent don't support HTTPS here
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

  # For a Apache running in a Docker container
  - id: "apache"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 80   # HTTP listener, Agent don't support HTTPS here
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

When using Docker, you may use Docker labels to set http_path and http_host:

docker run --labels glouton.http_path="/readiness" --labels glouton.http_host="my-host" [...]

Agent gathers the following metrics:

Metric	Description
service_status	Status of Apache
apache_busy_workers	Number of Apache worker busy
apache_busy_workers_perc	Apache worker busy in percent
apache_bytes	Network traffic sent by Apache in bytes per second
apache_connections	Number of client connections to Apache server
apache_idle_workers	Number of Apache workers waiting for an incoming request
apache_max_workers	Maximum number of Apache worker configured
apache_requests	Number of requests per second
apache_uptime	Time spent since Apache server start in seconds

Apache keeps track of server activity in a structure known as the scoreboard. There is a slot in the scoreboard for each worker and it contains the status of this worker. The size of the scoreboard is the maximum of concurrent users that Apache could server.

Metric	Description
apache_scoreboard_waiting	Number of workers waiting for an incoming request. It's the same as apache_idle_workers
apache_scoreboard_starting	Number of workers starting up
apache_scoreboard_reading	Number of workers reading an incoming request
apache_scoreboard_sending	Number of workers processing a client request
apache_scoreboard_keepalive	Number of workers waiting for another request via keepalive
apache_scoreboard_dnslookup	Number of workers looking up a hostname
apache_scoreboard_closing	Number of workers closing their connection
apache_scoreboard_logging	Number of workers writing in log files
apache_scoreboard_finishing	Number of workers gracefully finishing a request
apache_scoreboard_idle_cleanup	Number of idle workers being killed
apache_scoreboard_open	Number of slots with no worker in the scoreboard

Apache do not start all workers when they are not needed (e.g. if there is enough workers waiting, Apache reuse them and don't start a new one).

The sum of all scoreboard items is the maximum of concurrent requests Apache can serve. This sum is calculated and stored in apache_max_workers metric.

Asterisk

Service Detection	Specific Check	Metrics
✅	❌	❌

Bitbucket

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

In addition Bitbucket need to expose JMX over a TCP port. To enable JMX you can follow Enabling JMX counters for performance monitoring on Atlassian documentation.

Here is a summary for unauthenticated access:

Add option -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false to the JVM. Default Bitbucket startup-script accept them from JMX_OPTS or JAVA_OPTS environment variable.
Add jmx.enable=true to $BITBUCKET_HOME/shared/bitbucket.properties

Warning: this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

Bleemeo agent should auto-detect the JMX port, if some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Bitbucket running outside any container
  - id: "bitbucket"
    address: "127.0.0.1"
    port: 7990
    jmx_port: 3333
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For an additional Bitbucket running outside any container
  - id: "bitbucket"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 7991
    jmx_port: 3334
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For a Bitbucket running in a Docker container
  - id: "bitbucket"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 7990
    jmx_port: 3333
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

Agent gathers the following metrics:

Metric	Description
service_status	Status of Bitbucket
bitbucket_events	Number of events per second
bitbucket_io_tasks	Number of events per second
bitbucket_jvm_gc	Number of garbage collection per second
bitbucket_jvm_gc_utilization	Garbage collection utilization in percent
bitbucket_jvm_heap_used	Heap memory used in bytes
bitbucket_jvm_non_heap_used	Non-Heap memory used in bytes
bitbucket_pulls	Number of pulls per second
bitbucket_pushes	Number of pushes per second
bitbucket_queued_events	Number of events queued
bitbucket_queued_scm_clients	Number of SCM clients queued
bitbucket_queued_scm_commands	Number of SCM commands queued
bitbucket_request_time	Average time of request in seconds
bitbucket_requests	Number of requests per second
bitbucket_ssh_connections	Number of SSH connections per second
bitbucket_tasks	Number of scheduled tasks per second

Cassandra

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

In addition Cassandra need to expose JMX over a TCP port. To enable JMX you will need to:

If using Docker, add environment variable JVM_OPTS=-Dcassandra.jmx.remote.port=7199
If running Cassandra and Bleemeo agent as native package, nothing should be needed. Cassandra already expose JMX on port 7199 on localhost by default.
In other case, add option -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.authenticate=false to the JVM.

Warning: this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

Bleemeo agent should auto-detect the JMX port, if some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Cassandra running outside any container
  - id: "cassandra"
    address: "127.0.0.1"
    port: 9042
    jmx_port: 7199
    jmx_username: "cassandra"  # by default, no authentication is done
    jmx_password: "cassandra"

  # For an additional Cassandra running outside any container
  - id: "cassandra"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 9043
    jmx_port: 7200
    jmx_username: "cassandra"  # by default, no authentication is done
    jmx_password: "cassandra"

  # For a Cassandra running in a Docker container
  - id: "cassandra"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 9042
    jmx_port: 7199
    jmx_username: "cassandra"  # by default, no authentication is done
    jmx_password: "cassandra"

Agent gathers the following metrics:

Metric	Description
service_status	Status of Cassandra
cassandra_bloom_filter_false_ratio	Number of false positive of the bloom filter in percent
cassandra_jvm_gc	Number of garbage collection per second
cassandra_jvm_gc_utilization	Garbage collection utilization in percent
cassandra_jvm_heap_used	Heap memory used in bytes
cassandra_jvm_non_heap_used	Non-Heap memory used in bytes
cassandra_read_requests	Number of read requests per second
cassandra_read_time	Average time of read requests in seconds
cassandra_sstable	Number of SSTable
cassandra_write_requests	Number of write requests per second
cassandra_write_time	Average time of write requests in seconds

Bleemeo also support for detailed monitoring of specific Cassandra tables. To enable this, add the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Cassandra running outside any container
  - id: "cassandra"
    detailed_items:
      - "keyspace1.table1"
      - "keyspace2.table2"
    [...]

  # For a Cassandra running in a Docker container
  - id: "cassandra"
    instance: "CONTAINER_NAME"
    detailed_items:
      - "keyspace1.table1"
      - "keyspace2.table2"
    [...]

The following per-table metrics will be gathered:

Metric	Description
cassandra_bloom_filter_false_ratio	Number of false positive of the bloom filter in percent
cassandra_read_requests	Number of read requests per second
cassandra_read_time	Average time of read requests in seconds
cassandra_sstable	Number of SSTable
cassandra_write_requests	Number of write requests per second
cassandra_write_time	Average time of write requests in seconds

Confluence

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

In addition Confluence need to expose JMX over a TCP port. To enable JMX you can follow Live Monitoring Using the JMX Interface.

Here is a summary for unauthenticated access:

Add option -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false to the JVM. Default Confluence startup-script accept them from CATALINA_OPTS or JAVA_OPTS environment variable.

Warning: this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

Bleemeo agent should auto-detect the JMX port, if some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Confluence running outside any container
  - id: "confluence"
    address: "127.0.0.1"
    port: 8090
    jmx_port: 3333
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For an additional Confluence running outside any container
  - id: "confluence"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 8091
    jmx_port: 3334
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For a Confluence running in a Docker container
  - id: "confluence"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 8090
    jmx_port: 3333
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

Agent gathers the following metrics:

Metric	Description
service_status	Status of Confluence
confluence_db_query_time	Example for database query time in seconds
confluence_jvm_gc	Number of garbage collection per second
confluence_jvm_gc_utilization	Garbage collection utilization in percent
confluence_jvm_heap_used	Heap memory used in bytes
confluence_jvm_non_heap_used	Non-Heap memory used in bytes
confluence_last_index_time	Time of last indexing task in seconds
confluence_queued_error_mails	Number of mails in error queued
confluence_queued_index_tasks	Number of indexing tasks queued
confluence_queued_mails	Number of mails queued
confluence_request_time	Average time of request in seconds
confluence_requests	Number of requests per second

BIND

Service Detection	Specific Check	Metrics
✅	❌	❌

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a BIND running outside any container
  - id: "bind"
    address: "127.0.0.1"
    port: 53

  # For an additional BIND running outside any container
  - id: "bind"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 54

  # For a BIND running in a Docker container
  - id: "bind"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 53

Only the metric from service check is produced:

Metric	Description
service_status	Status of BIND

Dovecot

Service Detection	Specific Check	Metrics
✅	✅	❌

Agent uses a IMAP check if the service listen on port 143.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Dovecot running outside any container
  - id: "dovecot"
    address: "127.0.0.1"
    port: 143   # IMAP listener, Agent don't support IMAPS here

  # For an additional Dovecot running outside any container
  - id: "dovecot"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 144   # IMAP listener, Agent don't support IMAPS here

  # For a Dovecot running in a Docker container
  - id: "dovecot"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 143   # IMAP listener, Agent don't support IMAPS here

Only the metric from service check is produced:

Metric	Description
service_status	Status of Dovecot

ejabberd

Service Detection	Specific Check	Metrics
✅	❌	❌

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a ejabberd running outside any container
  - id: "ejabberd"
    address: "127.0.0.1"
    port: 5672

  # For an additional ejabberd running outside any container
  - id: "ejabberd"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 5673

  # For a ejabberd running in a Docker container
  - id: "ejabberd"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 5672

Only the metric from service check is produced:

Metric	Description
service_status	Status of ejabberd

Elasticsearch

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a HTTP check if the service listen on port 9200.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Elasticsearch running outside any container
  - id: "elasticsearch"
    address: "127.0.0.1"
    port: 9200

  # For an additional Elasticsearch running outside any container
  - id: "elasticsearch"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 9201

  # For a Elasticsearch running in a Docker container
  - id: "elasticsearch"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 9200

Agent gathers the following metrics:

Metric	Description
service_status	Status of Elasticsearch
elasticsearch_docs_count	Number of documents stored in all indices
elasticsearch_jvm_gc	Number of garbage collection per second
elasticsearch_jvm_gc_utilization	Garbage collection utilization in percent
elasticsearch_jvm_heap_used	Heap memory used in bytes
elasticsearch_jvm_non_heap_used	Non-Heap memory used in bytes
elasticsearch_size	Size of all indices in bytes
elasticsearch_search	Number of search in a shard per seconds
elasticsearch_search_time	Average time took by search in secondss
elasticsearch_cluster_docs_count	Number of documents stored in all indices of the cluster
elasticsearch_cluster_size	Size of all indices of the cluster in bytes

Exim4

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a SMTP check if the service listen on port 25.

To enable metrics gathering, ensure Bleemeo agent could run mailq command. This usually means to add the following to your exim configuration (e.g. /etc/exim4/conf.d/main/99_local):

queue_list_requires_admin=false

You will then need to fresh configuration (warning: the following will lost any local change in /etc/exim4/exim4.conf.template):

update-exim4.conf.template --run
update-exim4.conf
service exim4 restart

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Exim running outside any container
  - id: "exim"
    address: "127.0.0.1"
    port: 25

  # For a Exim running outside any container
  - id: "exim"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 26

  # For a Exim running in a Docker container
  - id: "exim"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 25

Agent gathers the following metrics:

Metric	Description
service_status	Status of Exim
exim_queue_size	Number of mails queued

Fail2ban

Service Detection	Specific Check	Metrics
✅	❌	✅

The agent will detect Fail2ban on your server and check that the server process stays active.

If you installed Glouton using wget or with a system package Glouton will gather metrics with no further configuration, otherwise, you need to allow Glouton to run fail2ban-client status as root. You can do so by adding the following to /etc/sudoers.d/glouton:

Cmnd_Alias FAIL2BAN = /usr/bin/fail2ban-client status, /usr/bin/fail2ban-client status *
glouton  ALL=(root) NOEXEC: NOPASSWD: FAIL2BAN
Defaults!FAIL2BAN !logfile, !syslog, !pam_session

The following metrics are gathered:

Metric	Description
service_status	Status of Fail2ban
fail2ban_failed	Number of failed authentications
fail2ban_banned	Number of banned IP

FreeRADIUS

Service Detection	Specific Check	Metrics
✅	❌	❌

HAProxy

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs access to HAProxy stats page and to know where it may find this page.

On HAProxy side, you will need to enable statistics page on a HTTP(S) frontend, for example:

frontend api-http
    bind 0.0.0.0:80
    stats enable
    stats uri     /statistics

On Bleemeo agent side, you need configure check by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a HAProxy running outside any container
  - id: "haproxy"
    address: "127.0.0.1"
    port: 80
    stats_url: "http://my-server/statistics"
    # For authenticated access, use
    # stats_url: "http://username:password@my-server/statistics"

  - id: "haproxy"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 81
    stats_url: "http://my-server:81/statistics"
    # For authenticated access, use
    # stats_url: "http://username:password@my-server:81/statistics"

  # For a HAProxy running in a Docker container
  - id: "haproxy"
    instance: "CONTAINER_NAME"
    address: "127.0.0.1"
    port: 80
    stats_url: "http://my-server/statistics"
    # For authenticated access, use
    # stats_url: "http://username:password@my-server/statistics"

Agent gathers the following metrics:

Metric	Description
service_status	Status of HAProxy
haproxy_act	Number of active servers
haproxy_bin	Network traffic received from clients in bytes per second
haproxy_bout	Network traffic sent to clients in bytes per second
haproxy_ctime	Average time spent opening a connection in seconds
haproxy_dreq	Number of request denied by HAproxy per second
haproxy_dresp	Number of response denied by HAproxy per second
haproxy_econ	Number of failed connection per second
haproxy_ereq	Number of request in error per second
haproxy_eresp	Number of response in error per second
haproxy_qcur	Number of currently queued requests
haproxy_qtime	Average time request spent in queue in seconds
haproxy_req_tot	Number of HTTP request per second
haproxy_rtime	Average response time in seconds
haproxy_scur	Number of sessions opened
haproxy_stot	Number of sessions per second
haproxy_ttime	Average time of session in seconds

InfluxDB

Service Detection	Specific Check	Metrics
✅	✅	❌

Agent uses a HTTP check if the service listen on port 8086.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a InfluxDB running outside any container
  - id: "influxdb"
    address: "127.0.0.1"
    port: 8086

  # For an additional InfluxDB running outside any container
  - id: "influxdb"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 8087

  # For a InfluxDB running in a Docker container
  - id: "influxdb"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 8086

Only the metric from service check is produced:

Metric	Description
service_status	Status of InfluxDB

Jenkins

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metric gathering, you need to configure the Jenkins URL and the credentials the agent should use to get information about the latest jobs.

info

To create a Jenkins API token, click your name on the top right, then click Configure and Add new Token.

To configure your service, add the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Jenkins running outside any container
  - id: "jenkins"
    address: "127.0.0.1"
    port: 8080
    # To enable metric gathering, the fields stats_url, username and password are required.
    # The other fields are optional and can be omitted.
    # Jenkins URL.
    stats_url: "http://jenkins.example.com"
    # Credentials used for authentication.
    username: my_user
    password: my_api_token
    ## TLS configurations.
    ca_file: "/myca.pem"
    cert_file: "/mycert.pem"
    key_file: "/mykey.pem"
    # Skip chain and host verification.
    ssl_insecure: false
    # Choose jobs to include or exclude. When using both lists, exclude has priority. 
    # Wildcards are supported: [ "jobA/*", "jobB/subjob1/*"]. If empty, all jobs are included.
    included_items: []
    excluded_items: []

  # For a Jenkins running in a Docker container
  - id: "jenkins"
    instance: "CONTAINER_NAME"
    port: 8080
    stats_url: "http://jenkins.example.com"
    username: my_user
    password: my_api_token

When using Docker, you may use labels to set the configuration:

docker run --labels glouton.stats_url="http://jenkins.example.com" --labels glouton.username="my_user" [...]

The following metrics are gathered:

Metric	Description
service_status	Status of Jenkins
jenkins_busy_executors	Number of busy executors
jenkins_total_executors	Total number of executors (both busy and idle)
jenkins_job_duration_seconds	Job duration in seconds
jenkins_job_number	Number of times this job has been run
jenkins_job_result_code	Job result code (0 = SUCCESS, 1 = FAILURE, 2 = NOT_BUILD, 3 = UNSTABLE, 4 = ABORTED)

JIRA

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

In addition Confluence need to expose JMX over a TCP port. To enable JMX you need to:

Add option -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false to the JVM. Default JIRA startup-script accept them from CATALINA_OPTS or JAVA_OPTS environment variable.

Warning: this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

Bleemeo agent should auto-detect the JMX port, if some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a JIRA running outside any container
  - id: "jira"
    address: "127.0.0.1"
    port: 8080
    jmx_port: 3333
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For an additional JIRA running outside any container
  - id: "jira"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 8081
    jmx_port: 3334
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For a JIRA running in a Docker container
  - id: "jira"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 8080
    jmx_port: 3333
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

Agent gathers the following metrics:

Metric	Description
service_status	Status of JIRA
jira_jvm_gc	Number of garbage collection per second
jira_jvm_gc_utilization	Garbage collection utilization in percent
jira_jvm_heap_used	Heap memory used in bytes
jira_jvm_non_heap_used	Non-Heap memory used in bytes
jira_request_time	Average time of request in seconds
jira_requests	Number of requests per second

Kafka

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

In addition, Kafka needs to expose JMX over a TCP port.

If you are using Docker, add the environment variable KAFKA_JMX_PORT=1234, see the documentation for details.

In other cases, you need to export some environment variable before running kafka-server-start.sh:

export JMX_PORT=1234
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1"

Warning: this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

The Bleemeo agent should auto-detect the JMX port, if some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Kafka running outside any container
  - id: "kafka"
    address: "127.0.0.1"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For an additional Kafka running outside any container
  - id: "kafka"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

  # For a Kafka running in a Docker container
  - id: "kafka"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 9092
    jmx_port: 1099
    jmx_username: "monitorRole" # by default, no authentication is done
    jmx_password: "secret"

The following metrics are gathered:

Metric	Description
service_status	Status of Kafka
kafka_jvm_gc	Number of garbage collection per second
kafka_jvm_gc_utilization	Garbage collection utilization in percent
kafka_jvm_heap_used	Heap memory used in bytes
kafka_jvm_non_heap_used	Non-Heap memory used in bytes
kafka_topics_count	Total number of topics
kafka_fetch_requests_sum	Total number of fetch requests per second
kafka_fetch_time_average	Average time to process a fetch request in secondss
kafka_produce_requests_sum	Total number of produce requests per second
kafka_produce_time_average	Average time to process a produce request in secondss

Bleemeo also supports detailed monitoring of specific Kafka topics. To enable this, add the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Kafka running outside any container
  - id: "kafka"
    detailed_items:
      - "topic1"
      - "topic2"
    [...]

  # For a Kafka running in a Docker container
  - id: "kafka"
    instance: "CONTAINER_NAME"
    detailed_items:
      - "topic1"
      - "topic2"
    [...]

When using Docker, you may use labels to set the detailed topics:

docker run --labels glouton.detailed_items="topic1,topic2" [...]

The following per-topic metrics will be gathered:

Metric	Description
kafka_fetch_requests	Number of fetch requests per second
kafka_produce_requests	Number of produce requests per second

If you want to enable more JMX metrics, you can add custom JMX metrics. A list of all the available metrics is available here.

libvirtd

Service Detection	Specific Check	Metrics
✅	❌	❌

Memcached

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a Memcached check if the service listen on port 11211.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Memcached running outside any container
  - id: "memcached"
    address: "127.0.0.1"
    port: 11211

  # For a Memcached running outside any container
  - id: "memcached"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 11212

  # For a Memcached running in a Docker container
  - id: "memcached"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 11211

Agent gathers the following metrics:

Metric	Description
service_status	Status of Memcached
memcached_command_get	Number of get request per second
memcached_command_set	Number of set request per second
memcached_connections_current	Number of client connection to Memcached
memcached_items_current	Current number of items stored
memcached_octets_rx	Network traffic received by Memcached in bytes per second
memcached_octets_tx	Network traffic sent by Memcached in bytes per second
memcached_ops_cas_hits	Number of successful CAS (Check-And-Set) request per second
memcached_ops_cas_misses	Number of CAS (Check-And-Set) request against missing keys per second
memcached_ops_decr_hits	Number of successful decr request per second
memcached_ops_decr_misses	Number of decr request against missing keys per second
memcached_ops_delete_hits	Number of successful delete request per second
memcached_ops_delete_misses	Number of delete request against missing keys per second
memcached_ops_evictions	Number of valid items removed from cache to free memory for new items per second
memcached_ops_get_hits	Number of successful get request per second
memcached_ops_get_misses	Number of get request against missing keys per second
memcached_ops_incr_hits	Number of successful incr request per second
memcached_ops_incr_misses	Number of incr request against missing keys per second
memcached_ops_touch_hits	Number of successful touch request per second
memcached_ops_touch_misses	Number of touch request against missing keys per second
memcached_percent_hitratio	Hit ratio of get request in percent
memcached_ps_count_threads	Number of worker threads
memcached_uptime	Time spent since Memcache server start in seconds

MongoDB

Service Detection	Specific Check	Metrics
✅	❌	✅

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a MongoDB running outside any container
  - id: "mongodb"
    address: "127.0.0.1"
    port: 27017

  # For a MongoDB running outside any container
  - id: "mongodb"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 27018

  # For a MongoDB running in a Docker container
  - id: "mongodb"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 27017

Agent gathers the following metrics:

Metric	Description
service_status	Status of MongoDB
mongodb_open_connections	Number of client connections to Mongo server
mongodb_net_in_bytes	Network traffic received by MongoDB in bytes per second
mongodb_net_out_bytes	Network traffic sent by MongoDB in bytes per second
mongodb_queued_reads	Number of clients waiting to read data from the MongoDB
mongodb_queued_writes	Number of clients waiting to write data to the MongoDB
mongodb_active_reads	Number of clients performing read operation
mongodb_active_writes	Number of clients performing write operation
mongodb_queries	Number of queries per second

Mosquitto

Service Detection	Specific Check	Metrics
✅	❌	❌

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a Mosquitto running outside any container
  - id: "mosquitto"
    address: "127.0.0.1"
    port: 1883  # MQTT listener, Agent don't support MQTT-SSL here

  # For an additional Mosquitto running outside any container
  - id: "mosquitto"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 1884  # MQTT listener, Agent don't support MQTT-SSL here

  # For a Mosquitto running in a Docker container
  - id: "mosquitto"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 1883  # MQTT listener, Agent don't support MQTT-SSL here

Only the metric from service check is produced:

Metric	Description
service_status	Status of Mosquitto

MySQL

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, credentials are required. Bleemeo agent will find the credentials if:

MySQL and the agent are running on Ubuntu or Debian
MySQL is running in a Docker container and root password is set through the environment variable MYSQL_ROOT_PASSWORD

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
[...]

  # For a MySQL running outside any container
  - id: "mysql"
    username: "USERNAME"
    password: "PASSWORD"
    address: "127.0.0.1"
    # If set to existing socket file, Glouton will prefer using the Unix socket to
    # connect and gather metric from MySQL.
    # Service check will continue to use the TCP address.
    metrics_unix_socket: "/var/run/mysqld/mysqld.sock"
    port: 3306

  # For an additional MySQL running outside any container
  - id: "mysql"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    username: "USERNAME"
    password: "PASSWORD"
    address: "127.0.0.1"
    # If set to existing socket file, Glouton will prefer using the Unix socket to
    # connect and gather metric from MySQL.
    # Service check will continue to use the TCP address.
    metrics_unix_socket: "/var/run/mysqld/mysqld2.sock"
    port: 3307

  # For a MySQL running in a Docker container
  - id: "mysql"
    instance: "CONTAINER_NAME"
    username: "USERNAME"
    password: "PASSWORD"
    address: "172.17.0.2"
    port: 3306

Agent gathers the following metrics:

Metric	Description
service_status: Status of MySQL
mysql_cache_result_qcache_hits	Number of query cache hits per second
mysql_cache_result_qcache_inserts	Number of queries added to the query cache per second
mysql_cache_result_qcache_not_cached	Number of uncacheable queries per second
mysql_cache_result_qcache_prunes	Number of queries that were deleted from the query cache because of low memory per second
mysql_cache_blocksize_qcache	Number of blocks in the query cache
mysql_cache_free_blocks	Number of free memory blocks in the query cache
mysql_cache_free_memory	Number of free memory for the query cache in bytes
mysql_cache_size_qcache	Number of queries registered in the query cache
mysql_locks_immediate	Number of table locks that could be granted immediately per second
mysql_locks_waited	Number of table locks that could not be granted immediately per second
mysql_innodb_history_list_len	Size of InnoDB transaction history list
mysql_innodb_locked_transaction	Number of InnoDB transaction currently locked
mysql_octets_rx	Network traffic received from clients in bytes per second
mysql_octets_tx	Network traffic sent to clients in bytes per second
mysql_queries	Number of queries per second
mysql_slow_queries	Number of slow queries per second
mysql_threads_cached	Number of threads in the thread cache
mysql_threads_connected	Number of currently open connections
mysql_threads_running	Number of threads that are not sleeping
mysql_total_threads_created	Number of threads created per second
mysql_commands_begin	Number of "BEGIN" statement executed per second
mysql_commands_binlog	Number of "BINLOG" statement executed per second
mysql_commands_call_procedure	Number of "CALL PROCEDURE" statement executed per second
mysql_commands_change_master	Number of "CHANGE MASTER" statement executed per second
mysql_commands_change_repl_filter	Number of "CHANGE REPL FILTER" statement executed per second
mysql_commands_check	Number of "CHECK TABLE" statement executed per second
mysql_commands_checksum	Number of "CHECKSUM TABLE" statement executed per second
mysql_commands_commit	Number of "COMMIT" statement executed per second
mysql_commands_dealloc_sql	Number of "DEALLOCATE PREPARE" statement executed per second
mysql_commands_stmt_close	Number of "DEALLOCATE PREPARE" statement executed per second
mysql_commands_delete_multi	Number of "DELETE" on multiple table statement executed per second
mysql_commands_delete	Number of "DELETE" statement executed per second
mysql_commands_do	Number of "DO" statement executed per second
mysql_commands_execute_sql	Number of "EXECUTE" statement executed per second
mysql_commands_stmt_execute	Number of "EXECUTE" statement executed per second
mysql_commands_explain_other	Number of "EXPLAIN FOR CONNECTION" statement executed per second
mysql_commands_flush	Number of "FLUSH" statement executed per second
mysql_commands_ha_close	Number of "HA CLOSE" statement executed per second
mysql_commands_ha_open	Number of "HA OPEN" statement executed per second
mysql_commands_ha_read	Number of "HA READ" statement executed per second
mysql_commands_insert_select	Number of "INSERT ... SELECT" statement executed per second
mysql_commands_insert	Number of "INSERT" statement executed per second
mysql_commands_kill	Number of "KILL" statement executed per second
mysql_commands_preload_keys	Number of "LOAD INDEX INTO CACHE" statement executed per second
mysql_commands_load	Number of "LOAD" statement executed per second
mysql_commands_lock_tables	Number of "LOCK TABLES" statement executed per second
mysql_commands_optimize	Number of "OPTIMIZE" statement executed per second
mysql_commands_prepare_sql	Number of "PREPARE" statement executed per second
mysql_commands_stmt_prepare	Number of "PREPARE" statement executed per second
mysql_commands_purge_before_date	Number of "PURGE BEFORE DATE" statement executed per second
mysql_commands_purge	Number of "PURGE" statement executed per second
mysql_commands_release_savepoint	Number of "RELEASE SAVEPOINT" statement executed per second
mysql_commands_repair	Number of "REPAIR" statement executed per second
mysql_commands_replace_select	Number of "REPLACE SELECT" statement executed per second
mysql_commands_replace	Number of "REPLACE" statement executed per second
mysql_commands_reset	Number of "RESET" statement executed per second
mysql_commands_resignal	Number of "RESIGNAL" statement executed per second
mysql_commands_rollback_to_savepoint	Number of "ROLLBACK TO SAVEPOINT" statement executed per second
mysql_commands_rollback	Number of "ROLLBACK" statement executed per second
mysql_commands_savepoint	Number of "SAVEPOINT" statement executed per second
mysql_commands_select	Number of "SELECT" statement executed per second
mysql_commands_signal	Number of "SIGNAL" statement executed per second
mysql_commands_slave_start	Number of "START SLAVE" statement executed per second
mysql_commands_group_replication_start	Number of "START" for group replication statement executed per second
mysql_commands_stmt_fetch	Number of "STMT FETCH" statement executed per second
mysql_commands_stmt_reprepare	Number of "STMT REPREPARE" statement executed per second
mysql_commands_stmt_reset	Number of "STMT RESET" statement executed per second
mysql_commands_stmt_send_long_data	Number of "STMT SEND LONG DATA" statement executed per second
mysql_commands_slave_stop	Number of "STOP SLAVE" statement executed per second
mysql_commands_group_replication_stop	Number of "STOP" for group replication statement executed per second
mysql_commands_truncate	Number of "TRUNCATE" statement executed per second
mysql_commands_unlock_tables	Number of "UNLOCK TABLES" statement executed per second
mysql_commands_update_multi	Number of "UPDATE" on multiple table statement executed per second
mysql_commands_update	Number of "UPDATE" statement executed per second
mysql_commands_xa_commit	Number of "XA COMMIT" statement executed per second
mysql_commands_xa_end	Number of "XA END" statement executed per second
mysql_commands_xa_prepare	Number of "XA PREPARE" statement executed per second
mysql_commands_xa_recover	Number of "XA RECOVER" statement executed per second
mysql_commands_xa_rollback	Number of "XA ROLLBACK" statement executed per second
mysql_commands_xa_start	Number of "XA START" statement executed per second
mysql_commands_assign_to_keycache	Number of assign to keycache commands per second
mysql_handler_commit	Number of internal commit request per second
mysql_handler_delete	Number of rows deleted from tables per second
mysql_handler_write	Number of rows inserted per second
mysql_handler_update	Number of rows updated per second
mysql_handler_rollback	Number of transaction rollback requests given to a storage engine per second

NATS

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, you need to enable the NATS monitoring endpoint on your server, see NATS Server Monitoring for details.

By default, the agent assumes that the monitoring endpoint is enabled on port 8222, if your server uses another port, you can change it in the configuration.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
[...]

  # For a NATS running outside any container
  - id: "nats"
    address: "127.0.0.1"
    port: 4222
    stats_port: 8222

  # For an additional NATS running outside any container
  - id: "mysql"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 4222
    stats_port: 8222

  # For a NATS running in a Docker container
  - id: "mysql"
    instance: "CONTAINER_NAME"
    port: 4222
    stats_port: 8222

The following metrics are gathered:

Metric	Description
service_status	Status of NATS
nats_uptime	Time since the NATS server is started in nanoseconds
nats_routes	Number of registered routes
nats_slow_consumers	Number of slow consumers
nats_subscriptions	Number of active subscriptions
nats_in_bytes	Amount of incoming bytes
nats_out_bytes	Amount of outgoing bytes
nats_in_msgs	Number of incoming messages
nats_out_msgs	Number of outgoing messages
nats_connections	Number of currently active clients
nats_total_connections	Total number of created clients

NFS

Service Detection	Specific Check	Metrics
✅	❌	✅

Metrics are gathered using /proc/self/mountstats.

NFS should be detected automatically on your server, but you can manually enable it by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
[...]
  - id: "nfs"

The following metrics are gathered:

Metric	Description
nfs_ops	The number of operations executed per second
nfs_transmitted_bits	The data exchanged in bits/s
nfs_rtt_per_op_seconds	The average round-trip time per operation in seconds
nfs_retrans	The number of times an operation had to be retried per second

Nginx

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a HTTP check if the service listen on port 80.

To enable metrics gathering, ensure Bleemeo agent can access nginx status on the URL http://server-address/nginx_status. This usually means you should add the following to your site definition (e.g. /etc/nginx/sites-enabled/default):

location /nginx_status {
    stub_status on;
}

If your nginx is not build with stub_status or if you need more information about nginx stub_status, see https://nginx.org/en/docs/http/ngx_http_stub_status_module.html

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a nginx running outside any container
  - id: "nginx"
    address: "127.0.0.1"
    port: 80   # HTTP listener, Agent don't support HTTPS here
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

  # For an additional nginx running outside any container
  - id: "nginx"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 81   # HTTP listener, Agent don't support HTTPS here
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:81"   # Host header sent. Like other option, you can omit them and default value will be used.

  # For a nginx running in a Docker container
  - id: "nginx"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 80   # HTTP listener, Agent don't support HTTPS here
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

When using Docker, you may use Docker labels to set http_path and http_host:

docker run --labels glouton.http_path="/readiness" --labels glouton.http_host="my-host" [...]

Agent gathers the following metrics:

Metric	Description
service_status	Status of nginx
nginx_requests	Number of requests per second
nginx_connections_accepted	Number of client connections established per second
nginx_connections_handled	Number of client connections processed per second
nginx_connections_active	Number of client connections to nginx server
nginx_connections_waiting	Number of idle client connections waiting for a request
nginx_connections_reading	Number of client connections where nginx is reading the request header
nginx_connections_writing	Number of client connections where nginx is writing the response

NTP

Service Detection	Specific Check	Metrics
✅	✅	❌

Agent uses a NTP check if the service listen on port 123.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

  # For a NTP running outside any container
  - id: "ntp"
    address: "127.0.0.1"
    port: 123

  # For a NTP running outside any container
  - id: "ntp"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 124

  # For a NTP running in a Docker container
  - id: "ntp"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 123

Only the metric from service check is produced:

Metric	Description
service_status	Status of NTP

OpenLDAP

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, you need to enable the slapd monitoring backend, and to set a bind CN and password in the agent configuration so it can access the metrics.

If some auto-detected parameters are wrong or you want to configure metric gathering, you can add the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a OpenLDAP running outside any container
  - id: "openldap"
    address: "127.0.0.1"
    port: 389
    # dn/password to bind with. If username is empty, an anonymous bind is performed.
    username: "cn=admin,dc=example,dc=org"
    password: "adminpassword"
    # Use ldaps, note that port will likely need to be changed to 636.
    ssl: true
    # Use StartTLS (note that you can't enable both ssl and starttls at the same time).
    starttls: true
    # Don't verify host certificate.
    ssl_insecure: true
    # Path to PEM-encoded Root certificate to use to verify the server certificate.
    ca_file: "/myca"

  # For an additional OpenLDAP running outside any container
  - id: "openldap"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 390
    username: "cn=admin,dc=example,dc=org"
    password: "adminpassword"

  # For a OpenLdap running in a Docker container
  - id: "openldap"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 389
    username: "cn=admin,dc=example,dc=org"
    password: "adminpassword"

The following metrics are gathered:

Metric	Description
service_status	Status of OpenLDAP
openldap_connections_current	Current number of active connections
openldap_waiters_read	Number of threads blocked waiting to read data from a client
openldap_waiters_write	Number of threads blocked waiting to write data to a client
openldap_threads_active	Threads (operations) currently active in slapd
openldap_statistics_bytes	Outgoing bytes per second
openldap_statistics_entries	Outgoing entries per second
openldap_operations_add_completed	Number of add operations per second
openldap_operations_bind_completed	Number of bind operations per second
openldap_operations_delete_completed	Number of delete operations per second
openldap_operations_modify_completed	Number of modify operations per second
openldap_operations_search_completed	Number of search operations per second

OpenVPN

Service Detection	Specific Check	Metrics
✅	❌	❌

Only the metric from service check is produced:

Metric	Description
service_status	Status of OpenVPN

PHP-FPM

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, Bleemeo agent needs access to PHP-FPM status page and to know where it may find this page.

PHP-FPM needs to expose its status. For example the pool configuration should include:

pm.status_path = /status

By defaults, Bleemeo agent try fcgi://<fpm-address>:<fpm-port>/status. E.g using FCGI over the TCP port on which PHP-FPM listen and use "/status" path.

If it does not match your configuration, you will need to override "stats_url" parameters by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a PHP-FPM running outside any container
  - id: "phpfpm"
    address: "127.0.0.1"
    port: 9000
    stats_url: "fcgi://127.0.0.1:9000/status"
    # For UNIX socket access, use
    # stats_url: "/var/run/php5-fpm.sock"
    # See below for additional note on UNIX socket permission

  # For an additional PHP-FPM running outside any container
  - id: "phpfpm"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 9001
    stats_url: "fcgi://127.0.0.1:9001/status"
    # For UNIX socket access, use
    # stats_url: "/var/run/php5-fpm2.sock"
    # See below for additional note on UNIX socket permission


  # For a PHP-FPM running in a Docker container
  - id: "phpfpm"
    instance: "CONTAINER_NAME"
    port: 9000
    stats_url: "fcgi://my-server:9000/status"

If using UNIX socket, you should make sure Glouton user has permission to access the socket. This usually means that Glouton user should be member of the group running php-fpm, which is www-data on Debian/Ubuntu system. Therefor sudo adduser glouton www-data and a restart of Bleemeo agent will grant access.

Agent gathers the following metrics:

Metric	Description
service_status	Status of PHP-FPM
phpfpm_accepted_conn	Number of requests per second
phpfpm_active_processes	Number of active processes
phpfpm_idle_processes	Number of idle processes
phpfpm_listen_queue	Number of requests in the queue of pending connections
phpfpm_listen_queue_len	Size of the queue of pending connections
phpfpm_max_active_processes	Maximum number of active processes since FPM started
phpfpm_max_children_reached	Number of times the process limit has been reached
phpfpm_max_listen_queue	Maximum number of requests in the queue of pending connections since FPM started
phpfpm_slow_requests	Number of slow requests per second
phpfpm_start_since	Time spent since PHP-fpm start in seconds
phpfpm_total_processes	Number of idle + active processes

Postfix

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a SMTP check if the service listen on port 25.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]

   # For a Postfix running outside any container
  - id: "postfix"
    address: "127.0.0.1"
    port: 25

   # For an additional Postfix running outside any container
  - id: "postfix"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 26

   # For a Postfix running in a Docker container
  - id: "postfix"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 25

Agent gathers the following metrics:

Metric	Description
service_status	Status of Postfix
postfix_queue_size	Number of mails queued

PostgreSQL

Service Detection	Specific Check	Metrics
✅	❌	✅

To enable metrics gathering, credentials are required. The agent will find the credentials by itself if PostgreSQL is running in a Docker container and username/password are set through the environment variables POSTGRES_USER (which defaults to postgres) and POSTGRES_PASSWORD.

By default, only sum metrics are gathered, if you want to monitor specific databases you can add them to the detailed_items setting.

If some auto-detected parameters are wrong, or you want to monitor specific databases, you can add the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a PostgreSQL running outside any container
  - id: "postgresql"
    username: "USERNAME"
    password: "PASSWORD"
    address: "127.0.0.1"
    port: 5432

  # For an additional PostgreSQL running outside any container
  - id: "postgresql"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    username: "USERNAME"
    password: "PASSWORD"
    address: "127.0.0.1"
    port: 5433

  # For a PostgreSQL running in a Docker container.
  - id: "postgresql"
    instance: "CONTAINER_NAME"
    username: "USERNAME"
    password: "PASSWORD"
    address: "172.17.0.2"
    port: 5432
    # Monitor the databases called "bleemeo" and "postgres".
    detailed_items:
      - bleemeo
      - postgres

When using Docker, you may use labels to set the databases to monitor and the credentials:

docker run --labels glouton.detailed_items="bleemeo,postgres" --labels glouton.username="USERNAME" --labels glouton.password="PASSWORD" [...]

Agent gathers the following metrics:

Metric	Description
service_status	Status of PostgreSQL
postgresql_blk_read_utilization	PostgreSQL reading data file blocks utilization
postgresql_blk_read_utilization_sum	Sum of PostgreSQL reading data file blocks utilization
postgresql_blk_write_utilization	PostgreSQL writing data file blocks utilization
postgresql_blk_write_utilization_sum	Sum of PostgreSQL writing data file blocks utilization
postgresql_blks_hit_sum	Number of blocks read from PostgreSQL cache per second
postgresql_blks_read_sum	Number of blocks read from disk per second
postgresql_commit_sum	Number of commits per second
postgresql_rollback_sum	Number of rollbacks per second
postgresql_temp_bytes_sum	Temporary file write throughput in bytes per second
postgresql_temp_files_sum	Number of temporary files created per second
postgresql_tup_deleted_sum	Number of rows deleted per second
postgresql_tup_fetched_sum	Number of rows fetched per second
postgresql_tup_inserted_sum	Number of rows inserted per second
postgresql_tup_returned_sum	Number of rows returned per second
postgresql_tup_updated_sum	Number of rows updated per second

Except for the status metric, all metrics also exist without the _sum suffix if detailed_items is used. For instance, the metric postgresql_commit with the item mycontainer_mydb corresponds to the number of commits per second on the database mydb running in the container mycontainer.

RabbitMQ

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a AMQP check if the service listen on port 5672.

To enable metrics gathering, credentials are required. By default "guest/guest" is used.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a RabbitMQ running outside any container
  - id: "rabbitmq"
    username: "USERNAME"
    password: "PASSWORD"
    address: "127.0.0.1"
    port: 5672          # Port of AMQP service
    mgmt_port: 15672    # Port of RabbitMQ management interface

  # For an additional RabbitMQ running outside any container
  - id: "rabbitmq"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    username: "USERNAME"
    password: "PASSWORD"
    address: "127.0.0.1"
    port: 5673          # Port of AMQP service
    mgmt_port: 15673    # Port of RabbitMQ management interface

  # For a RabbitMQ running in a Docker container
  - id: "rabbitmq"
    instance: "CONTAINER_NAME"
    username: "USERNAME"
    password: "PASSWORD"
    address: "172.17.0.2"
    port: 5672          # Port of AMQP service
    mgmt_port: 15672    # Port of RabbitMQ management interface

Agent gathers the following metrics:

Metric	Description
service_status	Status of RabbitMQ
rabbitmq_connections	Number of client connections to RabbitMQ server
rabbitmq_consumers	Number of consumers
rabbitmq_messages_acked	Number of messages acknowledged per second
rabbitmq_messages_count	Number of messages
rabbitmq_messages_delivered	Number of messages delivered per second
rabbitmq_messages_published	Number of messages published per second
rabbitmq_messages_unacked_count	Number of messages waiting for an acknowledgement from a consumer
rabbitmq_queues	Number of queues

Redis

Service Detection	Specific Check	Metrics
✅	✅	✅

Agent uses a Redis check if the service listen on port 6379.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a Redis running outside any container
  - id: "redis"
    address: "127.0.0.1"
    port: 6379
    password: "password-if-configured"

  # For an additional Redis running outside any container
  - id: "redis"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 6380
    password: "password-if-configured"

  # For a Redis running in a Docker container
  - id: "redis"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 6379
    password: "password-if-configured"

Agent gathers the following metrics:

Metric	Description
service_status	Status of Redis
redis_current_connections_clients	Number of connected clients
redis_current_connections_slaves	Number of connected slaves
redis_evicted_keys	Total number of key evicted keys due to maxmemory limit per second
redis_expired_keys	Total number of key expiration events per second
redis_keyspace_hits	Number of successful lookup of keys per second
redis_keyspace_misses	Number of successful lookup of keys per second
redis_keyspace_hitrate	Hit ratio of keys lookup in percent
redis_memory	Memory allocated by Redis in bytes
redis_memory_lua	Memory used by Lua engine in bytes
redis_memory_peak	Peak memory consumed by Redis in bytes
redis_memory_rss	Memory used by Redis as seen by system in bytes
redis_pubsub_channels	Global number of pub/sub channels with client subscriptions
redis_pubsub_patterns	Global number of pub/sub pattern with client subscriptions
redis_total_connections	Number of connections (client or slave) per second
redis_total_operations	Number of commands processed by the server per second
redis_uptime	Time spent since Redis server start in seconds
redis_volatile_changes	Number of changes since the last dump

Salt

Service Detection	Specific Check	Metrics
✅	❌	❌

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a Salt master running outside any container
  - id: "salt"
    address: "127.0.0.1"
    port: 4505

  # For an additional Salt master running outside any container
  - id: "salt"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 4505

  # For a Salt master running in a Docker container
  - id: "salt"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 4505

Only the metric from service check is produced:

Metric	Description
service_status	Status of Salt

Squid3

Service Detection	Specific Check	Metrics
✅	✅	❌

Agent uses a HTTP check if the service listen on port 3128.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a Squid running outside any container
  - id: "squid"
    address: "127.0.0.1"
    port: 3128
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

  # For an additional Squid running outside any container
  - id: "squid"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 3129
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:81"   # Host header sent. Like other option, you can omit them and default value will be used.

  # For a Squid running in a Docker container
  - id: "squid"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 3128
    http_path: "/"   # This is the path used for availability check, not metrics gathering.
    http_host: "127.0.0.1:80"   # Host header sent. Like other option, you can omit them and default value will be used.

When using Docker, you may use Docker labels to set http_path and http_host:

docker run --labels glouton.http_path="/readiness" --labels glouton.http_host="my-host" [...]

Only the metric from service check is produced:

Metric	Description
service_status	Status of Squid3

UPSD

Service Detection	Specific Check	Metrics
✅	❌	✅

The agent gathers metrics on UPS, you need to install and configure nut-server.

To configure NUT, please refer to the documentation.

You may need to provide the UPSD user credentials to the agent depending on your configuration.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  - id: "upsd"
    address: "127.0.0.1"
    port: 3128
    # UPSD user credentials.
    username: ""
    password: ""

The following metrics are gathered:

Metric	Description
service_status	Status of UPSD
upsd_battery_status	Battery status. The status is critical when the UPS is overloaded, on battery and when the battery is low or needs to be replaced.
upsd_status_flags	Status flags, see apcupsd for details.
upsd_battery_voltage	Battery voltage
upsd_input_voltage	Input voltage
upsd_output_voltage	Output voltage
upsd_load_percent	Load of the UPS in percent
upsd_battery_charge_percent	Battery charge in percent
upsd_internal_temp	Internal temperature in °C
upsd_input_frequency	Input frequency in Hz
upsd_time_left_seconds	Time left on battery in seconds
upsd_time_on_battery_seconds	Time spent on batttery in seconds

uWSGI

Service Detection	Specific Check	Metrics
✅	❌	✅

The agent gathers metrics using the uWSGI stats server, so you should enable it.

You can do so by adding the options --stats 127.0.0.1:1717 --memory-report to your server. Note that without --memory-report, the metric uwsgi_memory_used won't be available.

By default, the agent assumes that the stats server is enabled on port 1717, if your server uses another port, you can change it in the configuration.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a uWSGI running outside any container
  - id: "uwsgi"
    address: "127.0.0.1"
    port: 8080
    stats_port: 1717
    # If your server uses the --stats-http option, you should set the protocol to "http".
    # If the protocol is not set, "tcp" is used by default.
    stats_protocol: "tcp"

  # For an additional uWSGI running outside any container
  - id: "uwsgi"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 8080
    stats_port: 1717

  # For a uWSGI running in a Docker container
  - id: "uwsgi"
    instance: "CONTAINER_NAME"
    port: 8080
    stats_port: 1717

The following metrics are gathered:

Metric	Description
service_status	Status of uWSGI
uwsgi_requests	Number of requests per second
uwsgi_transmitted	Amount of data transmitted in bits/s
uwsgi_memory_used	Memory used in bytes
uwsgi_avg_request_time	Average time to process a request in seconds
uwsgi_exceptions	Number of expections per second
uwsgi_harakiri_count	Number of worker timeout per second

Varnish

Service Detection	Specific Check	Metrics
✅	❌	❌

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a Varnish running outside any container
  - id: "varnish"
    address: "127.0.0.1"
    port: 6082

  # For an additional Varnish running outside any container
  - id: "varnish"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 6083

  # For a Varnish running in a Docker container
  - id: "varnish"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 6082

Only the metric from service check is produced:

Metric	Description
service_status	Status of Varnish

ZooKeeper

Service Detection	Specific Check	Metrics
✅	✅	✅

The agent uses a ZooKeeper check if the service listen on port 2181.

The status check uses the ruok command, you may need to add it to the whitelist in 4lw.commands.whitelist, see the documentation for details.

To enable metrics gathering, Bleemeo agent needs to be installed with JMX enabled, see Java Metrics for setup details.

To gather JMX metrics, ZooKeeper must expose JMX over a TCP port. To enable JMX you will need to:

If using Docker for ZooKeeper, add environment variable JMXPORT=1234
In other cases, add the option -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false to the JVM. Usually in JAVA_OPTS in /etc/default/zookeeper.

Warning: in both case, this will allow unauthenticated access. Make sure no untrusted access to this port is possible or you should setup authenticated JMX access.

If some auto-detected parameters are wrong, you can manually override them by adding the following to /etc/glouton/conf.d/99-local.conf:

service:
  [...]
  # For a ZooKeeper running outside any container
  - id: "zookeeper"
    address: "127.0.0.1"
    port: 2181
    jmx_port: "JMX_PORT"
    jmx_username: "monitorRole"  # by default, no authentication is done
    jmx_password: "secret"

  # For an additional ZooKeeper running outside any container
  - id: "zookeeper"
    instance: "NAME_ASSOCIATED_WITH_YOUR_ADDITIONAL_SERVICE"
    address: "127.0.0.1"
    port: 2182
    jmx_port: "JMX_PORT"
    jmx_username: "monitorRole"  # by default, no authentication is done
    jmx_password: "secret"

  # For a ZooKeeper running in a Docker container
  - id: "zookeeper"
    instance: "CONTAINER_NAME"
    address: "172.17.0.2"
    port: 2181
    jmx_port: "JMX_PORT"
    jmx_username: "monitorRole"  # by default, no authentication is done
    jmx_password: "secret"

Agent gathers the following metrics:

Metric	Description
service_status	Status of ZooKeeper
zookeeper_connections	Number of client connections to ZooKeeper server
zookeeper_packets_received	Number of packets received sent per second
zookeeper_packets_sent	Number of packets sent per second
zookeeper_ephemerals_count	Number of ephemeral node
zookeeper_watch_count	Number of ZooKeeper watch
zookeeper_znode_count	Number of znode

Agent gathers the following metrics through JMX:

Metric	Description
zookeeper_jvm_gc	Number of garbage collection per second
zookeeper_jvm_gc_utilization	Garbage collection utilization in percent
zookeeper_jvm_heap_used	Heap memory used in bytes
zookeeper_jvm_non_heap_used	Non-Heap memory used in bytes

Services Monitoring

Common Features​

Service Status​

Ignore ports​

Docker labels​

Apache HTTP​

Asterisk​

Bitbucket​

Cassandra​

Confluence​

BIND​

Dovecot​

ejabberd​

Elasticsearch​

Exim4​

Fail2ban​

FreeRADIUS​

HAProxy​

InfluxDB​

Jenkins​

JIRA​

Kafka​

libvirtd​

Memcached​

MongoDB​

Mosquitto​

MySQL​

NATS​

NFS​

Nginx​

NTP​

OpenLDAP​

OpenVPN​

PHP-FPM​

Postfix​

PostgreSQL​

RabbitMQ​

Redis​

Salt​

Squid3​

UPSD​

uWSGI​

Varnish​

ZooKeeper​

Common Features

Service Status

Ignore ports

Docker labels

Apache HTTP

Asterisk

Bitbucket

Cassandra

Confluence

BIND

Dovecot

ejabberd

Elasticsearch

Exim4

Fail2ban

FreeRADIUS

HAProxy

InfluxDB

Jenkins

JIRA

Kafka

libvirtd

Memcached

MongoDB

Mosquitto

MySQL

NATS

NFS

Nginx

NTP

OpenLDAP

OpenVPN

PHP-FPM

Postfix

PostgreSQL

RabbitMQ

Redis

Salt

Squid3

UPSD

uWSGI

Varnish

ZooKeeper