Troubleshoot

Agent Files

Configuration

The agent reads its configuration files from:

• /etc/glouton/glouton.conf
• /etc/glouton/conf.d/*.conf
• etc/glouton.conf
• etc/conf.d/*.conf
• C:\ProgramData\glouton\glouton.conf
• C:\ProgramData\glouton\conf.d

The default installation creates the following files:

• /etc/glouton/glouton.conf: common default configuration and description of some customizable options.
• /etc/glouton/conf.d/05-system.conf: default options for integration with the system. For example, it includes the syslog logger.
• /etc/glouton/conf.d/30-install.conf: credentials used to communicate with the Bleemeo Cloud platform.

For more details on configuration files, see Configuration.

Diagnostic page

The Bleemeo agent has a built-in web server which provides a diagnostic page, available by default at http://localhost:8015/diagnostic

This page can help you find the issues you’re seeing.

Diagnostic Archive

A diagnostic archive, which contains more details (including recent debug log messages), is also available. This archive is primarily aimed at Bleemeo support and Bleemeo agent developers.

To retrieve the diagnostic.zip archive locally, run the following command:

TARGET_HOST={user}@{ip-of-your-server}
ssh $TARGET_HOST sh -c "'curl http://localhost:8015/diagnostic.zip || wget -O- http://localhost:8015/diagnostic.zip'" > diagnostic.zip

Logs

In case of trouble, the most valuable source of information is the log file.

Log messages may live in various locations, depending on how you run the agent:

• On Linux, when the agent is installed with a package or with the standard method, logs go to syslog (usually /var/log/syslog or /var/log/messages). You can also use journalctl -u glouton -f to follow the latest logs.
• On Windows, logs are usually in C:\ProgramData\glouton\logs.
• For Docker, use docker logs bleemeo-agent.

The log destination is set in the configuration files. For a system install, logging is set up in /etc/glouton/conf.d/05-system.conf:

logging:
  output: syslog

For more details, add the following to your configuration (/etc/glouton/conf.d/90-custom.conf) to increase log level to DEBUG:

logging:
  level: DEBUG

After the configuration change, the agent will reload automatically. You can force it to restart with systemctl restart glouton or docker restart glouton.

Duplicated agent

On each server, the agent should use its own credentials. If an agent detects that another agent is already connected to the Bleemeo Cloud platform using the same credentials, it will stop sending metrics to avoid overwriting data. It will also log an error message and send an email to the account managers to notify them of the problem.

Note

To detect that another agent is using the same credentials, the agent checks whether some server properties (such as the FQDN, MAC address, or even the Bleemeo agent PID) have been modified on the Bleemeo API by something other than the agent itself. The agent will report in its logs which properties changed.

There are three main cases where this can happen:

• Two agents are running on the same host. In this case you should stop one of them.
• You are migrating a server to new hardware. You should follow our migration guide.
• You copied the state to another server (for example because you cloned a server through AMI or server image creation). On the server where the state was copied, you should stop the agent, then remove both state files, and restart the agent.

Note

If you want to create a cloud image, please follow the installation for cloud image creation.

State

The agent is stateful. It keeps some information specific to the server it runs on — for example, its registration ID, the metrics seen, and the metrics registered with the Bleemeo Cloud platform.

There are two state files:

• a static state file that stores static information on the agent (like its credentials for the Bleemeo Cloud platform). It is usually stored in /var/lib/glouton/state.json.
• a cache state file that stores the cache of the detected services and metrics. It is usually stored in /var/lib/glouton/state.cache.json.

Memory locking

Glouton needs credentials to gather the metrics of some software (e.g., PostgreSQL or vSphere), and for keeping them safe, they’re stored in locked memory. As a consequence, the lockable memory limit set for the Glouton process should be large enough to handle that behavior. A limit of 8MB should be enough for most use cases.

If the current limit isn’t enough, a warning message like this one will be logged:

The amount of lockable memory (64 kB) may be insufficient, and should be at least 192 kB.

On Unix systems, ulimit can be defined with:

• systemd: add LimitMEMLOCK=8M:8M to the [Service] section of the unit file
• docker: add the option --ulimit memlock=8388608:8388608 to the docker run command
• docker compose: add the following section to the Glouton service:

  ulimits:
    memlock:
      soft: 8388608
      hard: 8388608