Skip to main content

💡 SquirrelDB Times Series Database Features

Pre-aggregation​

To improve queries over large period of time, SquirrelDB does some pre-aggregation of the points.

Metrics older than a day are pre-aggregated. This aggregation resamples the time series with a lower resolution (5 minutes) to make PromQL range queries over historical data more efficient.

The raw data is kept and the pre-aggregated data is only used when querying large ranges. Queries with small ranges still use the full resolution metrics.

The pre-aggregation processing supports throttling to reduce stress on Cassandra and SquirrelDB (see cassandra.aggregate.intended_duration)

caution

The pre-aggregation processing for the previous day starts at 8h UTC, this allows to ingest data up to 8 hours old without any backfilling issue.

If you backfill historical data, you will need to manually trigger the pre-aggregation with the /debug/preaggregate endpoint. It takes two query parameters, from and to that contains the dates between which the aggregation should be done, with the format YYYY-MM-DD.

Note that redoing a pre-aggregation that has already been done will cause a small disk increase on Cassandra.

Force pre-aggregated data​

You can force a query to use pre-aggregated data instead of raw points by setting the HTTP header X-SquirrelDB-ForcePreAggregated to true.

Force raw data​

Force using raw data instead of pre-aggregated points by setting the HTTP header X-SquirrelDB-ForceRaw to true. If both ForcePreAggregated and ForceRaw are true, ForceRaw has priority.

Metric expiration​

By default, metrics written to SquirrelDB expire after one year, this can be configured with cassandra.default_time_to_live.

You can choose the Time To Live of metrics written with the HTTP header X-SquirrelDB-TTL. The value is in seconds.

For instance, a metric written with the header X-SquirrelDB-TTL: 3600 will expire after one hour.

Multi tenancy​

SquirrelDB fully supports multi tenancy.

To use it, you need to provide the HTTP header X-SquirrelDB-Tenant to all you queries, read and write requests.

When reading data, the header will limit the results to only the metrics written by this tenant.

When writing data, the header will make all metrics written associated to this tenant. In practice, this means that a tenant label (__account_id by default) will be added to all metrics. This label can be modified with the setting tenant_label_name.

For example the header X-SquirrelDB-Tenant: tenant1 will limit all requests to the tenant named tenant1.

By default, SquirrelDB allows queries that don't provide the tenant header, if you want to change this behavior, you can enable require_tenant_header.

Limiting results​

HTTP headers can be used when reading data to limit the number of points and series a query will return.

X-SquirrelDB-Max-Evaluated-Series​

Limit the number of series that can be evaluated by this request. This overrides the promql.max_evaluated_series configuration.

Example: X-SquirrelDB-Max-Evaluated-Series: 100

X-SquirrelDB-Max-Evaluated-Points​

Limit the number of points that can be evaluated by this request. This overrides the promql.max_evaluated_points configuration.

Example: X-SquirrelDB-Max-Evaluated-Points: 100

X-SquirrelDB-Forced-Matcher​

Add one matcher to limit the evaluated series.

Example: X-SquirrelDB-Forced-Matcher: job=node_exporter

Mutable labels​

SquirrelDB supports mutable labels, they are labels that are never written to the store, they can only be used in PromQL queries and are converted to non mutable labels. This allows labels to be dynamically modified without rewriting all metrics.

Creating a mutable label can be useful to create server groups.

For instance, if we have many web servers, instead of writing a complex PromQL query every time we want to get the metrics from all our web servers, we can create a mutable label. For instance, if the metrics from two servers have the labels instance="web1" and instance="web2", we can create a mutable label group="web" that will select our web servers.

Create mutable labels​

SquirrelDB is designed to be multi-tenant, so each tenant has its own mutable labels.

To create the mutable labels, SquirrelDB exposes the API endpoints /mutable/names and /mutable/values.

/mutable/names allows creating a mutable label name associated to a standard label. To create the group label, you need to POST the following data:

[
  {
    "tenant": "tenant1",
    "name": "group",
    "associated_name": "instance"
  }
]

Then, /mutable/values allows creating values for the labels we created. To create the web group value, you need to POST the following data:

[
  {
    "tenant": "tenant1",
    "name": "group",
    "value": "web",
    "associated_values": ["web1", "web2"]
  }
]

Using mutable labels​

To use mutable labels in PromQL queries, SquirrelDB needs to know which tenant is associated to your query. For that you need to provide the tenant HTTP header in your requests.

For instance, to get the cpu_used of all our web servers, we can send the PromQL query cpu_used{group="web",__account_id="tenant1"}.

Delete mutable labels​

To delete the web group, you can send DELETE request on /mutable/values with this body:

[
  {
    "tenant": "tenant1",
    "name": "group",
    "value": "web"
  }
]

Or if you want to delete the group mutable label completely, you can use a DELETE request on /mutable/names with this data:

[
  {
    "tenant": "tenant1",
    "name": "group"
  }
]