SquirrelDB Troubleshooting

SquirrelDB exposes several debug endpoints. A running instance serves the list of the endpoints it supports, each with a description, on /debug/ — including the ones its current backend adds. That page is the authoritative list; what follows highlights the ones you are most likely to need.

curl http://squirreldb:9201/ready answers Ready once SquirrelDB has started and its storage is available.

Debugging a query

To understand what a specific query costs, send it with the X-SquirrelDB-Query-Debug header: SquirrelDB then logs what it did to answer it, including the request duration. X-SquirrelDB-Query-Verbose-Debug logs more.

curl -H 'X-SquirrelDB-Query-Debug: 1' \
    'http://squirreldb:9201/api/v1/query?query=cpu_used'

/debug/toggle_debug_query enables the same tracing for every query, until you call it again. It can be very verbose, so prefer the header when you can reproduce the query.

For more context around a problem, lower log.level to 0 (debug) or -1 (trace).

Flushing the short term store

Points are first written to the short term store (Redis or memory) and only later to the long term storage. /debug/flush forces that write immediately.

curl http://squirreldb:9201/debug/flush

Index

The endpoints below inspect the index of the Cassandra backend. You need to understand a few concepts to read their output:

shards are used to distribute index data among Cassandras, a shard corresponds to a week of data. The shards are only used in the index.
a posting corresponds to a label name and value; it lets you get all metrics where a label is present.

Index info

You can check the global index state on /debug/index_info.

It supports the URL parameter metricID: ?metricID=12: provides information on the metric with the ID 12.

Index dump

You can see all known metrics on /debug/index_dump. The dumps are CSV, with the columns metricID, labels, expirationDate.

You may want to dump the metrics by labels, expiration, shard or posting:

/debug/index_dump_by_labels?query=cpu_used{instance="argon:8015"}: metrics matching the labels in the query. start and end (format YYYY-MM-DD) optionally limit the search to a time range, which defaults to 1971 to now
/debug/index_dump_by_expiration?date=2006-01-02: metrics that might expire at a given date
/debug/index_dump_by_shard?shard_time=2006-01-02: metrics in given shard
/debug/index_dump_by_posting?name=disk_used&value=/: metrics in given posting. With name alone, every metric that has a label with that name, whatever its value
/debug/index_dump_by_posting?shard_time=2006-01-02&name=disk_used: metrics in given postings and shard

The special value shard_time=0001-01-01 queries the global shard, which holds SquirrelDB’s internal postings. Remember to quote the URL in your shell:

curl 'http://squirreldb:9201/debug/index_dump_by_posting?shard_time=0001-01-01&name=__global__all|metrics__'

Index verification

/debug/index_verify checks the index for inconsistencies and reports them. It accepts several parameters, each enabled by giving it any non-empty value:

fix: repair the inconsistencies found instead of only reporting them
lock: take the global new-metric lock for the duration of the run, so concurrent writes cannot produce false positives
strict: also report expirations and metric creations that are merely suspicious
now=2006-01-02: run the verification as if it were that date

curl 'http://squirreldb:9201/debug/index_verify?lock=1'

Blocking index writes

/debug/index_block stops writes to the Cassandra index for 5 minutes — calling it again extends the block. /debug/index_unblock releases it early. This exists to reproduce and observe the behaviour of SquirrelDB when the index is unavailable; it is not something to run on a healthy production instance.

Mutable labels

/debug/mutable_dump exports every known mutable label as CSV, and /debug/mutable_import replaces all of them with the content you send. Together they are the way to back up, restore or bulk-edit mutable labels:

curl http://squirreldb:9201/debug/mutable_dump > mutable-labels.csv
curl 'http://squirreldb:9201/debug/mutable_import?force' --data-binary @mutable-labels.csv

Profiling

The Go pprof endpoints are served under /debug/pprof/, for CPU profiles, heap profiles and goroutine dumps.

ClickHouse

When the ClickHouse backend is used, SquirrelDB exposes extra endpoints under /debug/clickhouse/.

Host health

GET /debug/clickhouse/host shows, for each address of clickhouse.addresses, whether the host is reachable (it answers a ping) and ready (it holds every expected table and its replica is not lagging), and why it is not when it is not. SquirrelDB routes queries only to ready hosts, so this is the first place to look when reads or writes fail or slow down.

The squirreldb_clickhouse_reachable_nodes and squirreldb_clickhouse_ready_nodes metrics expose the same information as counts.

Table creation

POST /debug/clickhouse/create_tables re-runs the schema creation (CREATE ... IF NOT EXISTS ON CLUSTER). SquirrelDB otherwise creates its tables only at first bootstrap, so this is what you call after recovering a lost ClickHouse node or after adding a server to a cluster.

Pre-aggregation

Both endpoints take from and to query parameters in RFC3339 format, and process the range in chunk windows (a Go duration, one week by default):

GET /debug/clickhouse/agg/diff compares the pre-aggregate to a fresh recomputation from the raw points, and reports how many buckets drifted.
POST /debug/clickhouse/agg/rebuild recomputes the pre-aggregate from the raw points.

Cluster migration

POST /debug/clickhouse/migrate_to_cluster turns the tables of a single-node ClickHouse into the replicated tables that clickhouse.cluster_name requires, keeping their data. It accepts dry_run=1 to only report the plan, and cleanup=1 to drop the emptied source tables. The whole procedure is described in growing a ClickHouse deployment.