Stack brief#
Get this stack| Service | Image | Role |
|---|---|---|
| grafana | grafana/grafana:13.1.0 | web UI and data source provisioning |
| clickhouse | clickhouse:26.6.1 | standalone analytical database and native metrics target |
| loki | grafana/loki:3.7.3 | log storage |
| mimir | grafana/mimir:3.1.2 | metrics storage |
| alloy | grafana/alloy:v1.17.1 | metrics scraper, log collector, and OTLP receiver |
| tempo | grafana/tempo:3.0.2 | application trace storage |
The main metrics path is ClickHouse -> Alloy -> Mimir -> Grafana.
This stack runs one authenticated ClickHouse server with the stackdb database and native HTTP, client, and Prometheus interfaces. Grafana reads ClickHouse and node metrics from Mimir, so query traffic, merges, parts, latency, storage, and host resources can be investigated from the same UI as server logs and application traces.
ClickHouse metrics flow#
ClickHouse exposes metrics, asynchronous metrics, profile events, errors, and histograms on its native Prometheus endpoint. Alloy scrapes clickhouse:9363 with job, service, and instance labels set to clickhouse and forwards the samples to Mimir. In the same process, Alloy collects node metrics, forwards Docker logs to Loki, accepts OTLP telemetry, and sends application traces to Tempo.
ClickHouse dashboard coverage#
The ClickHouse dashboard is organized around operational questions: whether the server is reachable, how much query and insert traffic it handles, whether merges and mutations keep up, which queries consume time and memory, how efficiently data is read and compressed, what server logs report, and whether the host has enough resources.
Server Health and Availability
Covers server status, uptime, table count, read-only state, restart detection, native metrics endpoint availability, and server identity.
Traffic, Queries, and Connections
Covers queries, selects, and inserts per second, active queries and connections, rows and bytes read or written, distributed activity, and failed-query rates.
Replication and Distributed Tables
Keeps the shared section but reports only the real single-server state. Cluster and shard variables are omitted, and replicated-table, replica-delay, queue, and distributed-connection series are not presented as an active cluster.
Keeper/ZooKeeper Coordination
Keeps the shared coordination section without claiming a Keeper quorum. Coordination sessions, leaders, latency, transactions, and errors remain absent or zero when Keeper or ZooKeeper is not configured.
Merges, Mutations, Parts, and Partitions
Covers active merges and mutations, background-pool work, merge throughput, MergeTree part growth, and part counts by partition.
Query Performance
Covers average query, SELECT, and INSERT latency, rows and bytes read, query memory, mark and range efficiency, mark-cache hits, and query-cache efficiency.
Slow Query Analytics
Covers slow reads, read back-off, external aggregation and sort spills, failed queries, and memory-limit failures.
Storage, Compression, and Retention
Covers disk and filesystem usage, MergeTree data and row growth, compressed reads, primary-key memory, temporary files, and disk I/O throughput.
ClickHouse Log Analytics
Covers log volume, severity-filtered views, errors and warnings, failed or slow queries, storage problems, and lifecycle events.
System and Container Resource Usage
Covers ClickHouse and host CPU, memory, network traffic, disk I/O, filesystem capacity, data-directory growth, load average, and restart detection.
The current ClickHouse dashboard uses raw PromQL and LogQL for ratios, rates, filters, and topology tokens. It is primarily time-series based, with stat summaries, a server table, explanatory notes, and live log views where those forms better fit the data.
Logs and traces#
ClickHouse writes its server log to container output, and Alloy forwards containers marked for observability to Loki with project, service, container, and container_id labels. The dashboard uses those streams for severity filters, failed-query investigation, storage warnings, and startup or shutdown events.
For OTLP traffic, Alloy exposes gRPC on 4317 and HTTP on 4318. OTLP logs are converted for Loki, OTLP metrics are converted to Prometheus remote write for Mimir, and OTLP traces are batched and sent to Tempo. Tempo is reserved for application traces; the ClickHouse dashboard does not depend on traces.
| Source | Destination |
|---|---|
| ClickHouse and Docker container logs | Loki |
| OTLP logs on 4317/4318 | Loki |
| OTLP metrics on 4317/4318 | Mimir |
| OTLP traces on 4317/4318 | Tempo |
| Node exporter metrics | Mimir |
| ClickHouse native Prometheus metrics | Mimir |