ClickHouse

Auto-Detection

Built-in Metrics

Prometheus Metrics

ClickHouse is a high-performance, column-oriented SQL database for real-time analytics.

Glouton can monitor ClickHouse in two ways:

• Auto-detection — recommended, minimal configuration. Glouton detects ClickHouse and collects a curated set of built-in metrics.
• Prometheus metrics — scrape ClickHouse’s native Prometheus endpoint for a fully customizable set of metrics.

Auto-Detection

Glouton detects ClickHouse by its process and listening port (default HTTP interface: 8123).

To gather metrics, Glouton queries ClickHouse’s system tables over the HTTP interface, which requires credentials. The user (default: default) must be allowed to read the system.* tables.

If the auto-detected parameters are incorrect, override them manually:

Linux

sudo tee /etc/glouton/conf.d/99-clickhouse.conf > /dev/null << 'EOF'
service:
  - type: "clickhouse"
    username: "default"
    password: "PASSWORD"
    address: "127.0.0.1"
    port: 8123
EOF

Windows

Create or edit C:\ProgramData\glouton\conf.d\99-clickhouse.conf:

service:
  - type: "clickhouse"
    username: "default"
    password: "PASSWORD"
    address: "127.0.0.1"
    port: 8123

Glouton automatically detects configuration changes.

Metric	Description
service_status	Status of ClickHouse
clickhouse_events_query	Number of queries executed per second
clickhouse_events_select_query	Number of SELECT queries executed per second
clickhouse_events_failed_query	Number of failed queries per second
clickhouse_events_query_time_seconds	Time spent executing queries per second, in seconds
clickhouse_events_mutation_time_seconds	Time spent applying mutations per second, in seconds
clickhouse_events_slow_read	Number of slow reads from the filesystem per second
clickhouse_events_network_receive_bytes	Bytes received over the network per second
clickhouse_events_network_send_bytes	Bytes sent over the network per second
clickhouse_metrics_active_query	Number of queries currently being executed
clickhouse_metrics_memory_tracking	Amount of memory currently allocated by the server, in bytes
clickhouse_metrics_delayed_inserts	Number of INSERT queries currently throttled because of too many parts
clickhouse_metrics_broken_disks	Number of disks currently in a broken state
clickhouse_metrics_broken_distributed_bytes_to_insert	Bytes of broken distributed data pending insertion
clickhouse_metrics_broken_distributed_files_to_insert	Number of broken distributed files pending insertion
clickhouse_mutations_running	Number of mutations currently running
clickhouse_mutations_failed	Number of failed mutations
clickhouse_processes_longest_running	Duration of the longest currently running query, in seconds
clickhouse_asynchronous_metrics_replicas_max_queue_size	Largest replication queue size across replicated tables

Prometheus Metrics

Service Configuration

ClickHouse exposes Prometheus metrics since version 20.1.2.4. Add a <prometheus> section to your configuration by editing config.xml or creating a drop-in file at /etc/clickhouse-server/config.d/prometheus.xml:

<clickhouse>
    <prometheus>
        <endpoint>/metrics</endpoint>
        <port>9363</port>
        <metrics>true</metrics>
        <events>true</events>
        <asynchronous_metrics>true</asynchronous_metrics>
    </prometheus>
</clickhouse>

Restart ClickHouse for the change to take effect. Metrics are then available at http://localhost:9363/metrics.

Glouton Configuration

Linux

sudo tee /etc/glouton/conf.d/99-clickhouse.conf > /dev/null << 'EOF'
metric:
  prometheus:
    targets:
      - url: "http://localhost:9363/metrics"
        name: "clickhouse"
        allow_metrics:
          - "ClickHouseProfileEvents_Query"
          - "ClickHouseMetrics_Query"
          - "ClickHouseAsyncMetrics_*"
          - "ClickHouseProfileEvents_InsertedRows"
          - "ClickHouseMetrics_MemoryTracking"
EOF

Windows

Create or edit C:\ProgramData\glouton\conf.d\99-clickhouse.conf:

metric:
  prometheus:
    targets:
      - url: "http://localhost:9363/metrics"
        name: "clickhouse"
        allow_metrics:
          - "ClickHouseProfileEvents_Query"
          - "ClickHouseMetrics_Query"
          - "ClickHouseAsyncMetrics_*"
          - "ClickHouseProfileEvents_InsertedRows"
          - "ClickHouseMetrics_MemoryTracking"

Glouton automatically detects configuration changes.

Customize collected metrics

The metrics below are our recommended defaults. You can adjust the allow_metrics list to include any metric from ClickHouse’s monitoring documentation, or use deny_metrics to exclude specific ones. See Metrics Filtering for details.

Metric	Description
ClickHouseProfileEvents_Query	Total number of queries executed by ClickHouse
ClickHouseMetrics_Query	Number of queries currently being executed
ClickHouseAsyncMetrics_*	Asynchronous metrics including system-level resource usage
ClickHouseProfileEvents_InsertedRows	Total number of rows inserted into tables
ClickHouseMetrics_MemoryTracking	Amount of memory currently allocated by the server in bytes

Monitoring Troubleshooting

See Troubleshoot a Service Check or Missing Metrics for what applies to every service: finding the address and port Glouton really uses, what each check message means, and how to read the collection error — which does not appear in the agent logs at the default level.

The Service Check is not OK

The check only opens the TCP port — it sends no command — so it fails when the port is wrong (Connection refused), when a firewall drops the packets (connection timed out after 10 seconds), or when the configured address cannot be parsed (Invalid TCP address). A green check proves the port is open, and nothing more.

Metrics are Missing

A passing check says nothing about the metrics: they are collected separately.

Glouton queries the system.* tables over the HTTP interface — stats_url if set, otherwise stats_port, port, or 8123 — as the default user unless username says otherwise.

Reproduce what Glouton does, from the machine where the agent runs:

curl -u default: 'http://127.0.0.1:8123/?query=SELECT%20count()%20FROM%20system.metrics'

What you see	Cause	Fix
a protocol error	Glouton points at the native protocol port (9000) instead of the HTTP interface (8123)	Set stats_port: 8123, or stats_url
Code: 516. Authentication failed	Wrong credentials	Set username and password
Code: 497. Not enough privileges	The user cannot read the system tables	Grant SELECT on system.* to that user
a connection error	The HTTP interface is disabled	Enable http_port in the ClickHouse configuration