SquirrelDB Metrics
SquirrelDB exposes Prometheus metrics on /metrics.
| Metric name | Description |
|---|---|
squirreldb_api_requests_seconds |
Total processing time in seconds (including sending response to client) |
Batch keeps recent points in fast storage for efficient access and is in charge of writing batch of metrics to Cassandra. It is only used by the Cassandra backend; the ClickHouse backend uses a write-ahead log instead.
| Metric name | Description |
|---|---|
squirreldb_batch_background_seconds |
Total background processing time in seconds |
squirreldb_batch_conflict_write_memorystore_total |
Total number of write conflicts during purge of old points in the memory store |
squirreldb_batch_duplicated_points_total |
Total duplicated points eliminated by batcher during write request |
squirreldb_batch_flush_points_total |
Total points processed by the flush of the memory store |
squirreldb_batch_new_points_during_tsdb_write_total |
Total number of points arrived during TSDB write (which required a 2nd write in memory store) |
squirreldb_batch_non_owner_write_total |
Total number of metrics written to TSDB by a SquirrelDB not owner of the metric |
squirreldb_batch_requests_points_total |
Total points processed by write and read requests |
squirreldb_batch_requests_seconds |
Total processing time in seconds |
squirreldb_batch_takeover_total |
Total metrics taken over by this SquirrelDB, after the SquirrelDB owning them crashed |
squirreldb_batch_transfer_owner_total |
Total number of metrics whose ownership was transferred to another SquirrelDB. |
ClickHouse
Section titled “ClickHouse”Metrics of the ClickHouse backend. They are only
exposed when ClickHouse is used, as the main
backend or as the
best-effort extra_write_backend.
| Metric name | Description |
|---|---|
squirreldb_clickhouse_flush_seconds |
Duration of a batch flush write to ClickHouse |
squirreldb_clickhouse_flush_points_total |
Total points written to ClickHouse |
squirreldb_clickhouse_flush_errors_total |
Total batch flushes that failed |
squirreldb_clickhouse_flush_rows_per_batch |
Points sent to ClickHouse in a single INSERT block |
squirreldb_clickhouse_flush_metrics_per_batch |
Distinct series flushed to ClickHouse in a single INSERT block |
squirreldb_clickhouse_register_seconds |
Duration of the registration of new series in the index |
squirreldb_clickhouse_registered_series_total |
Total series registered or refreshed in the index |
squirreldb_clickhouse_seen_series_size |
Current number of series in the registration cache |
squirreldb_clickhouse_read_seconds |
Synchronous processing time of a read, by step |
squirreldb_clickhouse_read_series_total |
Total series resolved by a read |
squirreldb_clickhouse_read_source_total |
Reads by point source (raw or aggregated), the pre-aggregation routing decision |
squirreldb_clickhouse_query_wait_total |
Query acquisitions that had to wait for a concurrency slot, by kind (read/write) |
squirreldb_clickhouse_reachable_nodes |
Number of ClickHouse hosts that answer a ping |
squirreldb_clickhouse_ready_nodes |
Number of ClickHouse hosts that hold every expected table and whose replica is not lagging |
Write-ahead log
Section titled “Write-ahead log”The ClickHouse backend buffers incoming points in a write-ahead log (in Redis when
it is configured, in memory otherwise) that one elected instance, the drainer,
periodically writes to ClickHouse. See
clickhouse.wal_flush_delay
and
clickhouse.wal_soft_threshold.
| Metric name | Description |
|---|---|
squirreldb_clickhouse_wal_length |
Current number of points buffered in the write-ahead log |
squirreldb_clickhouse_wal_drainer_is_leader |
1 if this instance currently holds the drainer lease, else 0 |
squirreldb_clickhouse_wal_backpressure_total |
Writes slowed or blocked by backpressure, by kind (slow or block) |
squirreldb_clickhouse_wal_dropped_points_total |
Points dropped because a best-effort ClickHouse was unavailable |
squirreldb_clickhouse_wal_dropped_min_timestamp_seconds |
Oldest point timestamp ever dropped, the lower bound of the range to repair; 0 if none |
squirreldb_clickhouse_wal_dropped_max_timestamp_seconds |
Newest point timestamp ever dropped, the upper bound of the range to repair; 0 if none |
The three wal_dropped_* metrics are only ever non-zero while ClickHouse runs as
a best-effort dual-write target: they give the range to re-run
squirreldb migrate
over.
Cluster
Section titled “Cluster”Cluster is the component in charge of the communication between SquirrelDBs in a cluster.
| Metric name | Description |
|---|---|
squirreldb_cluster_message_processing_seconds |
Total processing time spent to send/receive messages |
The index provides efficient indexing of metrics by their labels.
| Metric name | Description |
|---|---|
squirreldb_index_cache_access_total |
Total number of access of each cache |
squirreldb_index_cache_size |
Total number of entries in the in-memory cache |
squirreldb_index_cassandra_queries_seconds |
Total processing time spent in Cassandra in seconds |
squirreldb_index_create_seconds |
Total metric creation time in seconds (excluding time to take the lock) |
squirreldb_index_expire_conflict_total |
Total number of conflicts in expiration updates |
squirreldb_index_expire_ghost_metrics_total |
Total number of “ghost” metrics deleted by the expiration task (partially created metrics) |
squirreldb_index_expire_lock_seconds |
Total processing time with the new-metric lock held by the task in seconds. This is a global lock across all SquirrelDBs. |
squirreldb_index_expire_metrics_delete_total |
Total number of metrics deleted by the expiration task |
squirreldb_index_expire_metrics_total |
Total number of metrics processed by the expiration task (deleted or not) |
squirreldb_index_expire_move_seconds |
Total processing time to move metrics ID from one expiration list to another in seconds |
squirreldb_index_expire_seconds |
Total processing time of the expiration task in seconds |
squirreldb_index_lookup_id_concurrent_new_total |
Total number of concurrent creation of the same metric |
squirreldb_index_lookup_id_new_total |
Total number of metrics created |
squirreldb_index_lookup_id_refresh_total |
Total lookup for metric ID that caused a refresh of TTL |
squirreldb_index_lookup_id_request_seconds |
Total lookup request for metric IDs (from labels) time in seconds |
squirreldb_index_lookup_labels_seconds |
Total lookup for labels (from ID) time in seconds |
squirreldb_index_search_metrics_seconds |
Total time spent searching for metrics (from labels selector) |
squirreldb_index_search_metrics_total |
Total number of metrics matching a search from labels selector |
squirreldb_index_update_postings_seconds |
Total postings updates time in seconds (including time to take the lock) |
squirreldb_index_updated_postings_total |
Total number of postings updated |
Locks is a component in charge of acquiring locks to prevent data races.
| Metric name | Description |
|---|---|
squirreldb_locks_cassandra_queries_seconds |
Total processing time spent in Cassandra in seconds |
squirreldb_locks_lock_pending |
Number of goroutines trying to acquire a lock |
squirreldb_locks_lock_seconds |
Total time spent to acquire a lock (including waiting time) in seconds |
squirreldb_locks_lock_success_total |
Total number of successfully acquired locks |
squirreldb_locks_unlock_seconds |
Total time spent to release a lock in seconds |
Memory store
Section titled “Memory store”The memory store holds the points that are buffered before being written to the long term storage, when Redis is not configured.
| Metric name | Description |
|---|---|
squirreldb_memorystore_metrics |
Count of metrics known by the memory store |
squirreldb_memorystore_points |
Count of points stored by the memory store |
Mutable labels
Section titled “Mutable labels”Mutable labels metrics.
| Metric name | Description |
|---|---|
squirreldb_mutable_cache_access_total |
Total number of access to the cache |
squirreldb_mutable_cache_size |
Number of entries stored in the cache |
SquirrelDB uses Redis for short term storage: the batches of points with the Cassandra backend, the write-ahead log with the ClickHouse one.
| Metric name | Description |
|---|---|
squirreldb_redis_operations_points_total |
Total points processed by Redis operation |
squirreldb_redis_operations_seconds |
Total processing time of Redis operations in seconds |
The TSDB provides long term storage. squirreldb_tsdb_ingested_points_total and
squirreldb_tsdb_write_commit_seconds measure the boundary between SquirrelDB and
its backend, so they are exposed whichever backend is used, and let Cassandra and
ClickHouse be compared on the same basis. The other metrics below are specific to
Cassandra; with the ClickHouse backend they are replaced by the
ClickHouse ones.
| Metric name | Description |
|---|---|
squirreldb_tsdb_aggregated_until_seconds |
Most recent timestamp for which all shards are aggregated |
squirreldb_tsdb_aggregation_seconds |
Total processing time spent for aggregating each shard |
squirreldb_tsdb_cassandra_queries_seconds |
Total processing time spent in Cassandra itself in seconds |
squirreldb_tsdb_ingested_points_total |
Total points written into the backend, whichever it is |
squirreldb_tsdb_requests_points_total |
Total points processed by persistent TSDB |
squirreldb_tsdb_requests_seconds |
Total processing time in seconds |
squirreldb_tsdb_write_commit_seconds |
Duration of a write commit to the backend, whichever it is |
Metrics are exposed by the Go Prometheus collector.
| Metric name | Description |
|---|---|
go_goroutines |
Number of goroutines that currently exist |
go_info |
Information about the Go environment |
go_memstats_alloc_bytes |
Number of bytes allocated and still in use |
go_memstats_heap_alloc_bytes |
Number of heap bytes allocated and still in use |
go_memstats_sys_bytes |
Number of bytes obtained from system |
go_threads |
Number of OS threads created |
Process
Section titled “Process”Metrics are exposed by the Prometheus process collector.
| Metric name | Description |
|---|---|
process_cpu_seconds_total |
Total user and system CPU time spent in seconds |
process_max_fds |
Maximum number of open file descriptors |
process_open_fds |
Number of open file descriptors |
process_resident_memory_bytes |
Resident memory size in bytes |
process_start_time_seconds |
Start time of the process since unix epoch in seconds |
process_virtual_memory_bytes |
Virtual memory size in bytes |
process_virtual_memory_max_bytes |
Maximum amount of virtual memory available in bytes |