Troubleshoot the Agent Connection

How the agent talks to Bleemeo

The agent only ever opens outbound connections, and it opens two of them. Both must be allowed by your firewall: one working without the other gives a half-broken agent, which is the most common source of confusion.

Channel	Destination	Carries	If it is blocked
HTTPS	api.bleemeo.com TCP 443	Registration, configuration, declaration of the discovered services and metrics	The agent never registers, so it never appears in the agent list
MQTT-SSL	mqtt.bleemeo.com TCP 8883	Metric points, logs, live processes, real-time commands from the platform	The agent registers and shows up in the agent list, but no data ever arrives: dashboards stay empty and the agent status stays unknown

An agent that appears in the panel but never reports a single metric is therefore, almost always, an agent whose MQTT connection is blocked. Port 443 is open on most networks while port 8883 is not, so an outbound rule that covers only HTTPS produces exactly this symptom.

You do not even have to log in to the server to confirm it. About a minute after failing to connect, the agent opens a test connection to the MQTT port itself, and reports the outcome through the HTTPS API — which still works — so the agent turns critical in the panel with:

Agent can't access Bleemeo MQTT on port 8883, is the port blocked by a firewall?

The diagnostic page then tells you exactly which layer fails, and why.

Read the diagnostic page

The agent runs a local web server which serves a diagnostic page, by default on http://localhost:8015/diagnostic.

Every time that page is loaded, the agent opens a fresh TCP and TLS connection to api.bleemeo.com:443 and to mqtt.bleemeo.com:8883, from the machine, container or pod where it runs. It is a live connectivity test, not a copy of an old status, which makes it the fastest way to confirm or rule out a firewall problem.

How to reach it

The agent listens on 127.0.0.1 only, so use that address rather than localhost: where localhost also resolves to ::1, a client that does not fall back to IPv4 — such as the BusyBox wget in the agent container — fails with Connection refused.

Linux

curl -s http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

On a server you only reach over SSH:

ssh {user}@{ip-of-your-server} curl -s http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

Windows

Invoke-RestMethod http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

Docker

The agent container runs with --net=host, so the page is reachable from the host itself:

curl -s http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

If you did not use the host network, query it from inside the container so the test runs in the same network namespace as the agent:

docker exec bleemeo-agent wget -qO- http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

Kubernetes

kubectl exec -n default ds/glouton -- wget -qO- http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

ds/glouton picks one pod, so this checks one node. Firewall rules often differ between node pools, so target the nodes you care about:

kubectl exec -n default "$(kubectl get pod -n default -l app=glouton \
  --field-selector spec.nodeName=<node-name> -o name | head -1)" \
  -- wget -qO- http://127.0.0.1:8015/diagnostic.txt/diagnostic.txt

A healthy agent

The connectivity lines of the diagnostic page look like this (excerpt):

Glouton is registered with Bleemeo with ID <your agent UUID>
Glouton is currently connected. Last report to Bleemeo at 2026-09-03T10:02:44+02:00
Glouton is able to establish TLS connection to api.bleemeo.com:443 (203.0.113.20:443)
Glouton is able to perform HTTP request to "https://api.bleemeo.com"
Glouton is able to establish TLS connection to mqtt.bleemeo.com:8883 (203.0.113.21:8883)

Four things to check: the agent is registered, it is currently connected, and it can establish a TLS connection to both api.bleemeo.com:443 and mqtt.bleemeo.com:8883. A missing or failing line is what you are looking for.

The firewall is blocking MQTT

Glouton is currently NOT connected. Last report to Bleemeo at 0001-01-01T00:00:00Z
Glouton is able to establish TLS connection to api.bleemeo.com:443 (203.0.113.20:443)
Glouton is able to perform HTTP request to "https://api.bleemeo.com"
Unable to open TCP connection to mqtt.bleemeo.com:8883: dial tcp 203.0.113.21:8883: i/o timeout
mqtt.bleemeo.com resolve to 203.0.113.21:8883: is your firewall blocking connection to 203.0.113.21:8883 on TCP port 8883

This is the textbook case. DNS resolves, HTTPS works, and the TCP handshake to port 8883 never completes — so the agent registered and is visible in the panel, but cannot send anything.

The wording of the error tells you how the traffic is being stopped:

• i/o timeout: the packets are silently dropped, which is what a firewall configured with a DROP/deny policy does. Nothing on the network answers.
• connect: connection refused: something actively rejects the connection (a REJECT rule, or a device answering with a TCP reset).
• connect: network is unreachable / no route to host: routing or a local policy stops the packet before it even leaves.

In all three cases the fix is the same: allow outbound TCP 8883 to mqtt.bleemeo.com. See firewall rules to allow.

DNS does not resolve

Unable to open TCP connection to mqtt.bleemeo.com:8883: dial tcp: lookup mqtt.bleemeo.com: no such host
Unable to resolve DNS name mqtt.bleemeo.com: lookup mqtt.bleemeo.com: no such host

The agent could not even turn the hostname into an address. Usual causes are a resolver that is itself unreachable (outbound UDP 53 filtered), an internal split-horizon DNS that does not forward external names, or a container with no working resolv.conf.

TCP works but TLS fails

The connection reaches port 8883 but the TLS handshake does not complete. By far the most common reason is the clock of the machine:

Glouton is NOT able to establish TLS connection to mqtt.bleemeo.com:8883 (10.0.0.12:41508): tls: failed to verify certificate: x509: certificate has expired or is not yet valid

A certificate is only valid inside a date range, so a machine whose clock is off by more than a few weeks rejects a perfectly good certificate. This is routine on boards with no battery-backed clock — a Raspberry Pi that starts the agent before NTP has corrected the time lands here every boot. Fix the clock, ideally with NTP, and the handshake succeeds. A clock that is off also stops the agent from sending data at all, see connection disabled by the platform.

The other possible message is a certificate the agent cannot build a trust chain for:

Glouton is NOT able to establish TLS connection to mqtt.bleemeo.com:8883 (10.0.0.12:41508): tls: failed to verify certificate: x509: certificate signed by unknown authority

This means the certificate presented is not the Bleemeo one, so a device on the path is intercepting TLS. It is a rare cause: an appliance doing that breaks a great many other connections at the same time, so it usually shows up as a site-wide problem rather than as a Bleemeo-only one. Check the system CA store first — a container image with an outdated or missing ca-certificates package produces the same error without any interception. If TLS really is intercepted, there are two ways out:

1. Recommended — exempt mqtt.bleemeo.com and api.bleemeo.com from TLS inspection. MQTT is not HTTP, and most inspection engines break the connection even when the certificate is accepted.
2. Make the agent trust the intercepting CA with bleemeo.mqtt.cafile. Note this bundle replaces the system CA store for the MQTT connection, so it must contain the full trust chain. For the HTTPS channel, install the CA in the operating system trust store instead.

Caution

bleemeo.mqtt.ssl_insecure disables certificate verification altogether. It exists for agent development, not as a workaround for an inspecting proxy in production.

Check from the server itself

The diagnostic page is the better test, because it runs from exactly where the agent runs. These commands are useful to hand over to a network team, or when the agent is not running at all.

Linux

getent hosts mqtt.bleemeo.com
openssl s_client -connect mqtt.bleemeo.com:8883 -servername mqtt.bleemeo.com < /dev/null

openssl is present on nearly every Linux install and tests both layers at once. It prints CONNECTED when the port is open, then the result of the certificate verification, which is what you want to read:

Verify return code: 0 (ok)

Anything else is the TLS problem the agent is hitting, with the same meaning as on the diagnostic page.

If you only need to know whether the port is open, use whichever of these is installed:

nc -zv mqtt.bleemeo.com 8883
curl -v --max-time 10 telnet://mqtt.bleemeo.com:8883

Windows

Resolve-DnsName mqtt.bleemeo.com
Test-NetConnection -ComputerName mqtt.bleemeo.com -Port 8883

TcpTestSucceeded : True means the port is open.

Docker

Run the test inside the agent container, so it goes through the same network namespace, DNS and rules as the agent:

docker exec bleemeo-agent nc -vz mqtt.bleemeo.com 8883

Kubernetes

kubectl exec -n default ds/glouton -- nc -vz mqtt.bleemeo.com 8883

If your cluster enforces egress NetworkPolicy objects, check them too: they apply to the agent pod even though it uses the host network on some setups.

Firewall rules to allow

Two outbound rules are enough, and no inbound rule is ever needed:

Direction	Source	Destination	Protocol / Port
Outbound	The monitored server	api.bleemeo.com	TCP 443
Outbound	The monitored server	mqtt.bleemeo.com	TCP 8883

A few things worth passing on to whoever manages the firewall:

• Prefer hostname-based rules. api.bleemeo.com and mqtt.bleemeo.com are stable, but the IP addresses behind them may change.
• The MQTT connection is long-lived. A device that closes idle sessions aggressively will make the agent reconnect in a loop; the agent sends a keep-alive every 45 seconds, so an idle timeout above two minutes is safe.
• Do not apply TLS inspection or application-layer filtering to port 8883. It is MQTT over TLS, not HTTPS.
• Return traffic must be allowed, which a stateful firewall does on its own.

Connections through a proxy

The two channels do not behave the same way with a proxy:

• The HTTPS channel is a standard Go HTTP client: it honours the HTTPS_PROXY, HTTP_PROXY and NO_PROXY environment variables.
• The MQTT channel is a raw TLS socket on port 8883. An HTTP or HTTPS proxy is not used for it at all, whatever those variables say. The only proxy form it can go through is a SOCKS5 proxy, declared in the all_proxy environment variable.

On a systemd install, environment variables are set with systemctl edit glouton:

[Service]
Environment="HTTPS_PROXY=http://proxy.example.com:3128"
Environment="all_proxy=socks5://proxy.example.com:1080"

Then run systemctl daemon-reload && systemctl restart glouton.

Note

If no SOCKS5 proxy is available, the network must let the agent reach mqtt.bleemeo.com:8883 directly. There is no HTTPS-only fallback: metric points travel over MQTT.

What the logs say about the connection

The agent runs a health check once a minute. At the default log level, it logs this for as long as the connection is down:

Bleemeo connection (MQTT) is currently not established

Most of the detail is one level below, so set logging.level to VERBOSE (DEBUG is mainly useful to Bleemeo developers) in /etc/glouton/conf.d/90-custom.conf:

logging:
  level: VERBOSE

The agent reloads on its own after the change. Then look for:

Log message	Meaning
Agent can't access Bleemeo MQTT on port 8883, is the port blocked by a firewall?	Logged at the default level, and pushed to the agent status shown in the panel
Unable to connect to Bleemeo MQTT (retry in 1m20s): ...	Each failed attempt, with the underlying network error — the same error the diagnostic page shows
Too many attempts to connect to Bleemeo MQTT were made in the last 10 minutes. Disabling MQTT for 5m3s	After 7 attempts in 10 minutes, the agent pauses for about 5 minutes before trying again
Bleemeo MQTT connection lost: ...	The connection was established and then dropped. Repeated over and over, it points at an idle timeout or a session-tracking device on the path
Bleemeo MQTT connection established	The connection is up
MQTT connection fail to re-establish since <date>. This may be a long network issue or a Glouton bug	The agent has been unable to reconnect for a long time

Retries use an exponential backoff, from 5 seconds up to 10 minutes, so a fresh glouton restart is the quickest way to see the error again right after changing a firewall rule.

See Logs for where the log messages are written on each platform.

Connection disabled by the platform

Beyond firewalls, the agent itself may suspend its connection. The diagnostic page then shows a line such as:

Glouton connection to Bleemeo is disabled until 2026-09-03T11:20:00+02:00 (14m37s remain) due to 'local time is too different from actual time'

Reason	What to do
local time is too different from actual time	Fix the clock of the server, ideally with NTP. The agent compares its clock with the Bleemeo API and refuses to send data that would be timestamped wrongly. The panel shows Agent local time too different from actual time on the agent status
duplicated state.json	Two agents share the same credentials, see duplicated agent
authentication error with Bleemeo API	Check bleemeo.account_id and bleemeo.registration_key in /etc/glouton/conf.d/30-install.conf
this agent is too old, and cannot be connected to our managed service	Upgrade the agent
too many errors, long standing error	Transient; the agent retries on its own. If it persists, collect a diagnostic archive
too many requests - client is throttled	The agent is being rate-limited and will resume by itself

Agent Files

Configuration

The agent reads its configuration files from:

• /etc/glouton/glouton.conf
• /etc/glouton/conf.d/*.conf
• etc/glouton.conf
• etc/conf.d/*.conf
• C:\ProgramData\glouton\glouton.conf
• C:\ProgramData\glouton\conf.d

The default installation creates the following files:

• /etc/glouton/glouton.conf: common default configuration and description of some customizable options.
• /etc/glouton/conf.d/05-system.conf: default options for integration with the system. For example, it includes the syslog logger.
• /etc/glouton/conf.d/30-install.conf: credentials used to communicate with the Bleemeo Cloud platform.

For more details on configuration files, see Configuration.

Diagnostic Archive

A diagnostic archive, which contains more details (including recent debug log messages), is also available. This archive is primarily aimed at Bleemeo support and Bleemeo agent developers.

To retrieve the diagnostic.zip archive locally, run the following command:

TARGET_HOST={user}@{ip-of-your-server}
ssh $TARGET_HOST sh -c "'curl http://127.0.0.1:8015/diagnostic.zip || wget -O- http://127.0.0.1:8015/diagnostic.zip'" > diagnostic.zip

Logs

In case of trouble, the most valuable source of information is the log file.

Log messages may live in various locations, depending on how you run the agent:

• On Linux, when the agent is installed with a package or with the standard method, logs go to syslog (usually /var/log/syslog or /var/log/messages). You can also use journalctl -u glouton -f to follow the latest logs.
• On Windows, logs are usually in C:\ProgramData\glouton\logs.
• For Docker, use docker logs bleemeo-agent.

The log destination is set in the configuration files. For a system install, logging is set up in /etc/glouton/conf.d/05-system.conf:

logging:
  output: syslog

For more details, add the following to your configuration (/etc/glouton/conf.d/90-custom.conf) to raise the log level. VERBOSE is enough for most issues, including connection problems; DEBUG is more verbose and mainly useful to Bleemeo support:

logging:
  level: VERBOSE

After the configuration change, the agent will reload automatically. You can force it to restart with systemctl restart glouton or docker restart glouton.

Duplicated agent

On each server, the agent should use its own credentials. If an agent detects that another agent is already connected to the Bleemeo Cloud platform using the same credentials, it will stop sending metrics to avoid overwriting data. It will also log an error message and send an email to the account managers to notify them of the problem.

Note

To detect that another agent is using the same credentials, the agent checks whether some server properties (such as the FQDN, MAC address, or even the Bleemeo agent PID) have been modified on the Bleemeo API by something other than the agent itself. The agent will report in its logs which properties changed.

There are three main cases where this can happen:

• Two agents are running on the same host. In this case you should stop one of them.
• You are migrating a server to new hardware. You should follow our migration guide.
• You copied the state to another server (for example because you cloned a server through AMI or server image creation). On the server where the state was copied, you should stop the agent, then remove both state files, and restart the agent.

Note

If you want to create a cloud image, please follow the installation for cloud image creation.

State

The agent is stateful. It keeps some information specific to the server it runs on — for example, its registration ID, the metrics seen, and the metrics registered with the Bleemeo Cloud platform.

There are two state files:

• a static state file that stores static information on the agent (like its credentials for the Bleemeo Cloud platform). It is usually stored in /var/lib/glouton/state.json.
• a cache state file that stores the cache of the detected services and metrics. It is usually stored in /var/lib/glouton/state.cache.json.

Memory locking

Glouton needs credentials to gather the metrics of some software (e.g., PostgreSQL or vSphere), and for keeping them safe, they’re stored in locked memory. As a consequence, the lockable memory limit set for the Glouton process should be large enough to handle that behavior. A limit of 8MB should be enough for most use cases.

If the current limit isn’t enough, a warning message like this one will be logged:

The amount of lockable memory (64 kB) may be insufficient, and should be at least 192 kB.

On Unix systems, ulimit can be defined with:

• systemd: add LimitMEMLOCK=8M:8M to the [Service] section of the unit file
• docker: add the option --ulimit memlock=8388608:8388608 to the docker run command
• docker compose: add the following section to the Glouton service:

  ulimits:
    memlock:
      soft: 8388608
      hard: 8388608