High Availability

Every component of the Bleemeo Community Edition can be scaled up and provide high availability.

Quickstart

A docker compose is available to start a monitoring stack with:

• a Cassandra cluster with 3 nodes
• a Redis cluster with 6 nodes
• 2 SquirrelDB behind a load balancer (nginx)
• a NATS cluster with 3 nodes
• 2 SquirrelDB Ingestors
• 2 Glouton

For true high availability, each component must run on a different node, which is is not done by the docker compose.

NATS

We recommend using NATS as the MQTT server used by Glouton because it supports clustering for high availability.

See the NATS clustering documentation to setup a NATS cluster.

SquirrelDB

SquirrelDB can be scaled up, and rely on Cassandra and Redis clusters for high availability.

See the SquirrelDB high availability documentation for details.

SquirrelDB Ingestor

SquirrelDB Ingestor can be scaled up with NATS subject mapping. It distributes the messages received on the data topic across several ingestors. Use deterministic partitioning: each agent is assigned to a partition based on a hash of its FQDN, so the points of a given server are always processed by the same ingestor and stay in order. This requires nats-server 2.9.5 or later.

For example to run two ingestors you can use the following NATS mapping configuration:

mappings = {
  # Distribute the agents across two partitions: "v1/agent/<fqdn>/data" is rewritten to
  # "v1/agent/<fqdn>/data/0" or "v1/agent/<fqdn>/data/1" based on a hash of the FQDN.
  v1.agent.*.data: "v1.agent.{{wildcard(1)}}.data.{{partition(2,1)}}"
}

Then run one ingestor per partition: pass --mqtt-id 0 to the first ingestor and --mqtt-id 1 to the second. Partitions are numbered from 0. The default MQTT client ID is suffixed with the --mqtt-id value, so both ingestors can share the same MQTT username.

Note that running multiple ingestors doesn’t give you high availability, if one ingestor is down, the messages won’t be processed until it’s restarted. This is not much of an issue since the messages are buffered in NATS so you won’t lose any metric. Moreover, the ingestor is stateless, so it can be restarted with the same configuration, which might be automated by an orchestrator like Kubernetes (the /healthz and /readyz endpoints of the ingestor can serve as the liveness and readiness probes).