Skip to content
BleemeoBleemeo

Troubleshoot a Service Check or Missing Metrics

How a Service is Monitored

Section titled “How a Service is Monitored”

A detected service is monitored by two independent mechanisms, and knowing which one is failing tells you where to look:

What it does What you see in Bleemeo
Service check Opens the service port at least once a minute, and for some services sends a protocol command and waits for the expected answer. Each attempt times out after 10 seconds The service_status metric and the status text on the service
Metric collection Connects as a client to the service’s own statistics interface (administration database, status page, management API…) The service’s built-in metrics, for example redis_memory or mysql_queries

They fail for different reasons, and the two are not tied together: a service can pass its check and still report no metric, or report metrics while its check is critical.

Between two runs, the check keeps a connection open to each address the service listens on, so a service that stops accepting connections turns critical within seconds rather than at the next minute. It also means a service listening on several addresses is only OK while all of them answer.

The page of each monitored service lists the errors specific to that service and how to fix them. This page covers what is common to all of them.

Find the Address and Port Glouton Uses

Section titled “Find the Address and Port Glouton Uses”

Everything below depends on one thing: the address and port Glouton believes your service listens on. Ask the agent directly, on the machine, container or pod where it runs:

Terminal window
curl -s http://127.0.0.1:8015/diagnostic.txt/discovery.txt

It lists one line per discovered service, with the address and port used by the check and by the metric collection:

# Discovered service (count=2, last update=2026-09-09T10:12:41Z)
myservice on IP 127.0.0.1, active=true, instance=, container=, listening on [127.0.0.1:1234/tcp], type=myservice
otherservice on IP 172.17.0.2, active=true, instance=CONTAINER_NAME, container=CONTAINER_NAME, listening on [172.17.0.2:5678/tcp], type=otherservice

Compare the listening on list with reality:

Terminal window
# from iproute2
sudo ss -lntp


# from net-tools, when ss is unavailable
sudo netstat -lntp


# from lsof, when neither is installed
sudo lsof -nP -iTCP -sTCP:LISTEN

Run whichever your system has, rather than chaining them. If sudo answers sudo: ss: command not found while the command exists, it is not in the secure_path of sudo — call it by its absolute path, /usr/sbin/ss.

If the two do not match, declare the correct values in a service configuration file — the snippet is on each service page:

Terminal window
sudo tee /etc/glouton/conf.d/99-myservice.conf > /dev/null << 'EOF'
service:
  - type: "myservice"
    address: "127.0.0.1"
    port: 1234
EOF

Glouton picks up the change on its own, without a restart.

Why Glouton Can Look at the Wrong Port

Section titled “Why Glouton Can Look at the Wrong Port”

Reading which process owns which listening socket requires root, and the agent does not run as root. On Linux it reads that information from a file instead: /var/lib/glouton/netstat.out, produced by glouton-netstat — a wrapper around netstat -lnp — which the packaging runs when the service starts, and once an hour from cron.

When that file is missing, empty or outdated, Glouton has no listening address for the process and falls back to the default port of the service type, which may not be the port you run. This is the single most common cause of a Connection refused on a service that is perfectly healthy: Glouton is knocking on a port nobody listens on. The port in the error message is the giveaway — if it is the default port of the service type rather than yours, the problem is the discovery, not the service.

Two things go wrong with that file:

  • netstat is not installed. net-tools is a recommended package, not a required one, so a system installed with --no-install-recommends or a minimal image has no netstat at all. glouton-netstat then fails silently and the file stays empty.
  • It is up to an hour old. A service that just moved to another port keeps its previous address until the next hourly run.

Check the file, and refresh it by hand:

Terminal window
sudo cat /var/lib/glouton/netstat.out
sudo /usr/sbin/glouton-netstat

If netstat is missing, install net-tools and run the command again. Setting address and port explicitly in the service configuration also works, and is the right answer when the agent cannot see the process at all — a service in another container or another network namespace.

The path of the file is configurable through agent.netstat_file.

The Service Check is not OK

Section titled “The Service Check is not OK”
Status text What happened Usual cause
TCP port N, Connection refused Nothing accepted the connection The service listens on another port or another address, or it is not running
TCP port N, connection timed out after 10 seconds The connection, or the answer, never came A firewall dropping the packets, or a service too busy to answer
TCP port N, unexpected response "..." The service answered, but not what the protocol expects The service requires an authentication Glouton does not have, or another service listens on that port
TCP port N, no data received from host The connection was accepted, then closed with no answer The service dropped the client, often because of an access control list
TCP port N, connection closed / connection closed too early The service hung up during the exchange TLS expected on a plain-text port, or a connection limit reached
Invalid TCP address "..." / Invalid TCP port "..." The configured address could not be parsed A typo in the address or port of the service configuration

For the services whose check goes beyond opening the port, the text of the unexpected response is the service’s own error message, quoted verbatim. Read it as such: it is usually far more specific than the check itself, and the service page lists the ones you are likely to hit.

Metrics are Missing

Section titled “Metrics are Missing”

A service can pass its check and still report only service_status. Glouton collects the built-in metrics as a client of the service’s own statistics interface, which usually has to be enabled or opened first. There is nothing to enable in Bleemeo: what is missing is on the service side, or in the Glouton configuration.

What it takes depends on the service. Find yours below, then open its page: what to enable, and where, is specific to each one and is documented there.

Your service What Glouton needs
Elasticsearch, Fail2ban, Memcached, MongoDB, NFS, NSQ, ZooKeeper Nothing on the agent side. The service itself may still refuse to answer, so the fix is on its own page
Apache, NATS, nginx, PHP-FPM, RabbitMQ, uWSGI A statistics interface enabled in the service. Its name and the way to turn it on differ for each — see its page
ClickHouse, HAProxy, Jenkins, MySQL / MariaDB, OpenBao, OpenLDAP, PgBouncer, PostgreSQL, Redis, Vault Credentials in the Glouton configuration. Without them there is no metric at all, and the check itself may be critical when the service needs authentication to answer it
Bitbucket, Cassandra, Confluence, Jira, Kafka A JMX port, and jmxtrans installed — see Java Monitoring with JMX

Metrics are also plan-dependent: built-in service metrics require the Professional plan, while the availability check is available on every plan. See the services overview for the details.

Read the Collection Error

Section titled “Read the Collection Error”

When Glouton has everything it needs to connect but the connection itself fails, the error comes straight from the service — and it is the single most useful thing to read.

Do not look for it in the agent logs: collection errors are logged one level below the default, so a healthy-looking log proves nothing. Ask the agent for its scrape results instead:

Terminal window
curl -s http://127.0.0.1:8015/diagnostic.txt/scrape-loop-active.json

Each collector appears once, a service input under the name <service> input. Its last runs are listed in LastScrape, and a failed one carries an Error holding the message the service returned:

{
  "Description": "redis input",
  "LastScrape": [
    {
      "ScrapeAt": "2026-09-15T15:18:30+02:00",
      "ScrapeDuration": "1.2ms",
      "ScrapedPointsCount": 0,
      "Error": "read from redis: NOAUTH Authentication required."
    }
  ]
}

A ScrapedPointsCount of 0 with no Error means the collector ran and the service answered nothing — usually a statistics interface that is reachable but switched off.

If you would rather follow the errors live, set logging.level to VERBOSE in /etc/glouton/conf.d/90-custom.conf; they then appear as Gather of metrics failed on <service> input: .... The agent reloads on its own, and repeated failures are rate-limited to one message per collector every few minutes.

Still Stuck

Section titled “Still Stuck”

Send us the diagnostic archive: it contains discovery.txt and scrape-loop-active.json along with the agent logs and configuration, which is everything we need to tell you what the agent saw. Contact support with it attached.