Guide
Health
At-a-glance cluster health dashboard with automated headless health-sweep alerts via webhook and Cloudflare Cron.
Get a single status grid of cluster health across replication, merges, errors, disks, load, and parts — plus a headless sweep endpoint that fires webhook alerts on a schedule.
Prop
Type
What it does
The Health page aggregates checks across multiple system tables into a single status grid. Each check has a severity level (ok / warning / critical) and a human-readable summary. Operators use this page as a starting point for incident triage.
Checks cover:
- Replication lag — replicas falling behind, readonly tables
- Merge backlog — active merge count and slow merges
- Error rate — recent errors from
system.errors - Disk usage — free space across all disks
- Query load — running query count and memory pressure
- Part health — excessive part counts
- TTL & Partition Health — MergeTree tables with partition bloat, a time-based
PARTITION BYand no table TTL, or a merge backlog (recommend-only; full inventory on/ttl-partition-health)
In addition to the UI, chmonitor exposes a headless health-sweep endpoint (GET /api/cron/health-sweep) that runs the same checks and dispatches webhook alerts. This is designed to be called on a schedule (e.g., Cloudflare Cron every 5 minutes) without a browser.
Pages
| Page | Route | What it shows | System tables |
|---|---|---|---|
| Health | /health | Status grid with per-check severity and details | system.metrics, system.asynchronous_metrics, system.replicas, system.merges, system.errors, system.disks, system.replication_queue, system.processes, system.query_log, system.parts, system.tables |
Using it
Open /health for the status grid — each check reports ok / warning / critical with a summary. Start incident triage here, then drill into the linked feature sections for detail.
To run the same checks without a browser, call the health-sweep endpoint (see below) on a schedule.
Alert Settings layout
/alert-settings has four tabs: Alerts, Thresholds, Activity, and Advanced. The Alerts tab answers "what alerts do I have, and where do they go?" on one screen:
- Built-in alerts — one row per health check, with its name and stable id.
- More ways to alert — Suggested alerts (thresholds derived from how this cluster behaved) and Custom rules (your own rule on any metric).
- Channels — browser, server delivery channels, and custom webhook targets.
- Delivery — Routing rules (which alerts go to which channel), Webhook subscriptions (POST alert events to your own endpoints), and Digest (batch non-critical alerts).
- Minimum severity — the default floor every channel inherits.
The Advanced tab now holds only Silencing: Quiet hours and Maintenance windows. Each item still opens in its own dialog. Old links keep working: ?tab=routing, ?tab=webhooks, ?tab=digest, ?tab=suggested, and ?tab=custom-rules open that dialog on the Alerts tab; ?tab=maintenance and ?tab=quiet-hours open it on Advanced. ?tab=define, ?tab=delivery, and ?tab=silencing open the tab that holds that group.
On the hosted cloud dashboard, quiet hours and maintenance windows are saved for your own connections, but scheduled alert checks do not cover your own connections yet. Until they do, these windows do not silence any alert, and both panels say so.
Naming built-in alerts
Each built-in alert can be renamed from the list (1–80 characters). The name is display-only in the dashboard: notifications still use the built-in check title, and the alert keeps its check id, so a rename never resets alert state or acknowledgements. Names are stored in the metadata database; with no database the list still renders, but rename is read-only.
Permissions & access
CHM_FEATURE_HEALTH_ACCESS=authenticatedCHM_FEATURE_HEALTH_ENABLED=false[features.health]
enabled = true
access = "authenticated"Configuration
Run alerts without a metadata database
Alerting works on Docker and Kubernetes with no D1 and no Postgres, but there is one boundary to learn up front: declarative config can fire alerts. It cannot acknowledge them.
| Data | Declarative? | Why |
|---|---|---|
| Alert definitions: rules, thresholds, routing, channels, webhook targets, quiet hours, maintenance windows, digest settings | Yes | Desired state. An operator declares intent. |
| Alert state: firing or cleared, hysteresis streak, incident start, last-notified | No | Observation, not intent. It cannot be authored. |
| ACKs, suggestion dismissals, digest buffer | No | Per-user runtime state. |
With no database, alert state lives in memory in each worker instance. So
hysteresis streaks and incident timers reset on every restart or deploy, and
Acknowledge, Clear and dismiss are disabled in the UI. POST and DELETE on
/api/v1/health/ack return 501. The ClickHouse state backend
(CHM_STATE_CLICKHOUSE_*) does not back the alert stores; the alert surface
runs on D1 or Postgres (DATABASE_URL / POSTGRES_URL). Attach one of them if
you need ACKs. See GET /api/v1/config capabilities.health
for what a deployment resolved.
In-memory alert state is not migrated when you attach a database later. On the first sweep that reaches the new database, the in-memory state is discarded and alerting starts fresh from the database: a pending hysteresis streak restarts, and a condition that is still firing opens a new incident and can notify again.
Where definitions come from
Definitions merge from four sources, highest first:
- D1 or Postgres rows, written from the UI.
- Config files in
CHM_HEALTH_CONFIG_DIRECTORY(default/etc/chmonitor/health.d). - Environment variables (
HEALTH_ALERT_*,HEALTH_THRESHOLD_*). - Built-in defaults.
Every source is always read; precedence only decides who wins a key. Per key:
- Most definitions (routes, custom rules, webhook targets, quiet hours, channels) merge field by field. A higher source overrides a field only when it sets a non-empty value, so a file can set
enabledwhile a D1 row owns a token. A higher source cannot clear a lower source's value back to empty. - Maintenance windows are replaced whole per
id, and windows from different sources are all active. - Digest settings are a single value: the highest source wins outright.
- A key that only a lower source defines disappears when that source stops defining it. There is no tombstone.
Declared entries are shared by every user (deployment-wide, like the env
fallbacks). In the UI they are read-only and show a Config file or Env
badge instead of Delete or Reset. Custom rules declared with id: foo appear as
custom:foo. The directory is read once per process, so a changed file needs
a pod restart. Secrets are read from the environment on each read.
The files
One YAML file per concern, named exactly <concern>.yaml (or .yml). Files with
other names are skipped with a warning.
| File | Top-level keys | Entry key |
|---|---|---|
alerts.yaml | rules, thresholds, peerdbRules, repeat | id (rules, peerdbRules), rule name (thresholds) |
routing.yaml | routes | id |
channels.yaml | channels, webhookTargets | channel, id |
quiet-hours.yaml | windows | id |
maintenance.yaml | windows | id |
digest.yaml | enabled, windowMinutes | none (single value) |
Every keyed entry carries its key explicitly. The schema is strict: unknown keys are rejected, so a typo is reported instead of silently ignored.
Secrets never go in these files. A ConfigMap is not a Secret. Secret fields
take the name of an environment variable (secretEnv, urlEnv,
headersEnv), which you back with a Kubernetes Secret. An unset variable skips
that entry with a warning that names the variable, never its value. Declared
outbound URLs must be HTTPS and pass the same SSRF check as the API. A URL that
is itself a credential (Slack or Discord webhook) belongs in a routes entry
secretEnv or a webhookTargets urlEnv, not in a channel target.url.
alerts.yaml declares custom rules. metric is a key from the rule
builder catalog (for example failed-mutations, readonly-replicas,
replication-max-lag, disk-usage-percent). op is one of >, >=, <,
<=, and warning and critical must be ordered for that operator.
rules:
- id: failed-mutations-any
name: Failed mutations
metric: failed-mutations
op: ">="
warning: 1
critical: 5alerts.yaml also accepts a thresholds: map (rule id to warning /
critical). The sweep and the current-findings snapshot apply it over the
built-in default. It is merged with the HEALTH_THRESHOLD_<RULE>_WARNING and
HEALTH_THRESHOLD_<RULE>_CRITICAL environment variables field by field, and
the file wins a collision: warning: 70 in the file with
HEALTH_THRESHOLD_DISK_USAGE_CRITICAL=95 in env gives 70 / 95. A rule id
that is not registered is skipped with a warning in the log. The file is read
once per process, so restart after editing it; env changes apply on the next
sweep. Only built-in rule ids are honoured by the current-findings snapshot.
thresholds:
disk-usage:
warning: 70alerts.yaml also accepts a repeat: block that paces reminders. It is the
file form of HEALTH_ALERT_COOLDOWN_WARNING_MINUTES /
HEALTH_ALERT_COOLDOWN_CRITICAL_MINUTES (cooldownMinutes),
HEALTH_ALERT_REPEAT_MIN_DELTA and HEALTH_ALERT_REPEAT_MIN_DELTA_<RULE>
(minDelta, minDeltaByRule), and HEALTH_ALERT_BACKOFF_MAX_MINUTES
(backoffMaxMinutes, must be above 0), and
HEALTH_ALERT_LONG_FIRING_DIGEST_HOURS (longFiringDigestHours, must be above
0; off by default). The last one sends one summary per day per channel of
alerts that have been open at least that many hours, with the first and the
current value. Every key is optional and the file
wins over env per key. A per-rule min delta (file, then env) still beats the
global one. An invalid block is skipped with a warning and env keeps applying.
repeat:
cooldownMinutes:
warning: 120
critical: 15
minDelta: 2
minDeltaByRule:
disk-usage: 10
backoffMaxMinutes: 240
longFiringDigestHours: 24alerts.yaml also accepts peerdbRules:, per-mirror PeerDB thresholds and
mutes. Each entry sets one check (lag, slot-lag, errors, stale-sync,
throughput-zero)
for the mirrors that match match (matchKind: exact, prefix, or glob,
the default). warning must not be above critical. severity: warning stops
that check from going past warning. muteUntil (ISO-8601) mutes the matching
mirrors until then. Rules from the file show as read-only in Alert Settings.
See PeerDB monitoring
for how matches are ranked.
peerdbRules:
- id: fleetreporting-lag
check: lag
match: qrep_sg_fleetreporting1_*
warning: 3600
critical: 14400
- id: orders-errors-quiet
check: errors
matchKind: exact
match: pg_orders
warning: 3
critical: 20
severity: warning
muteUntil: 2026-10-12T06:00:00Zrouting.yaml declares routes. provider is webhook, pagerduty,
telegram, ntfy, pushover or matrix; matchRule and matchHost default
to * and accept globs. secretEnv names the variable holding the webhook URL,
PagerDuty routing key, Telegram bot token, ntfy token (optional), Pushover app
token or Matrix access token. target carries the non-secret fields:
serviceName (pagerduty), chatId (telegram), url (ntfy), user (pushover),
and homeserverUrl plus roomId (matrix).
routes:
- id: oncall-pagerduty
provider: pagerduty
matchRule: "*"
matchHost: "*"
minSeverity: critical
secretEnv: CHM_ROUTE_PAGERDUTY_KEY
target:
serviceName: chmonitor-prod
- id: team-telegram
provider: telegram
matchRule: "custom:*"
secretEnv: CHM_ROUTE_TELEGRAM_TOKEN
target:
chatId: "-1001234567890"channels.yaml declares the server delivery channels (webhook,
healthchecks, email, opsgenie, telegram, ntfy, pushover, twilio) and
custom webhook targets. A target's urlEnv is required; headersEnv names a
variable holding a JSON object of secret headers. repeat: false sends the
target the first alert, any escalation and the recovery, but no reminders
(default true, reminders on). Precedence against env is per
channel, not per field.
channels:
- channel: telegram
minSeverity: warning
secretEnv: HEALTH_CHANNEL_TELEGRAM_TOKEN
target:
chatId: "-1001234567890"
- channel: opsgenie
secretEnv: HEALTH_CHANNEL_OPSGENIE_KEY
target:
region: eu
webhookTargets:
- id: ops-chat
name: Ops chat
format: slack
minSeverity: warning
urlEnv: CHM_WEBHOOK_OPS_CHAT_URL
headersEnv: CHM_WEBHOOK_OPS_CHAT_HEADERS
repeat: falsequiet-hours.yaml takes days (0 to 6), start and end as HH:mm, an
IANA timezone (default UTC) and an optional severityCap: critical.
windows:
- id: nightly
days: [1, 2, 3, 4, 5]
start: "22:00"
end: "07:00"
timezone: Asia/Ho_Chi_Minh
severityCap: criticalmaintenance.yaml takes an optional hostId (omit for all hosts), a
reason, and ISO-8601 startsAt / endsAt with endsAt later.
windows:
- id: upgrade-2026-10-05
hostId: 0
reason: ClickHouse 25.8 upgrade
startsAt: "2026-10-05T01:00:00Z"
endsAt: "2026-10-05T03:00:00Z"digest.yaml is a single object. windowMinutes is 0 to 1440.
enabled: true
windowMinutes: 15Fail-closed behaviour
- A missing directory, or a runtime with no filesystem (Cloudflare Workers), is a silent no-op.
- Bad YAML, unknown keys, schema violations, entries without a key and duplicate keys are skipped with a warning. The rest of the file and the other files still load, and the pod never fails to start.
- Warnings name the file and entry, never a field value, so a secret pasted into a ConfigMap by mistake does not reach the logs.
- An entry whose
*Envvariable is unset, or whose URL is not an allowed HTTPS endpoint, is skipped. - With no database and no declared custom rule, the rule builder still reports "not available" (
501) on write.
For the Kubernetes mount, see Alert definitions from a ConfigMap. Variable reference: CHM_HEALTH_CONFIG_DIRECTORY.
Health-sweep cron alerting
The health-sweep endpoint runs checks over all configured hosts and sends a webhook notification when a check meets or exceeds the minimum severity.
| Variable | Default | Description |
|---|---|---|
CRON_SECRET | (unset = endpoint disabled) | Required. Guards GET /api/cron/health-sweep and GET /api/cron/retention-prune. Pass as Authorization: Bearer <secret> only — query-string ?secret= is rejected because it lands in access logs. When it is unset/empty the endpoints fail closed and return HTTP 503 — they do not run. Note that retention-prune is destructive (it deletes conversation rows past each plan's retention window), which is why these routes refuse to run unauthenticated. |
CHM_HEALTH_SWEEP_ENABLED | (unset = enabled iff CRON_SECRET is set) | Gates the scheduled (headless) sweep fired by the Cloudflare Cron trigger. Set true to force-enable, false to pause scheduled sweeps without removing the cron. When unset it fails closed to the CRON_SECRET posture. Does not affect manual curl hits (those only need a valid CRON_SECRET). Truthy values: 1/true/yes/on. |
HEALTH_ALERT_ENABLED | (optional) | Master switch. Unset: alerts dispatch as soon as any channel is configured (webhook URL, email, PagerDuty, and so on); nothing configured stays in dry run. Set false (0/no/off) to pause all dispatch; true/1/yes/on forces it on. |
HEALTH_ALERT_WEBHOOK_URL | (unset = webhook off) | Incoming webhook URL. The channel is auto-detected from the URL: Slack (hooks.slack.com) and Discord (discord.com/api/webhooks) get channel-native rich bodies; Microsoft Teams (*.webhook.office.com, or a Workflows *.logic.azure.com URL) gets an Adaptive Card; Google Chat (chat.googleapis.com) gets a cardsV2 card; any other URL receives the plain {"text": "...", "content": "..."} wrapper. See Alerting to Slack and Discord for setup examples. |
HEALTH_ALERT_HEALTHCHECKS_URL | (unset = healthchecks disabled) | healthchecks.io ping URL. The sweep GETs it on each alert (bare URL) and appends /fail on recovery. Self-hosted or cloud healthchecks.io both work. Overridable from the Server delivery channels UI (see callout below). |
HEALTH_ALERT_MIN_SEVERITY | warning | Global default minimum severity that triggers a notification. Values: warning or critical. Per-channel and per-route overrides below take precedence. |
HEALTH_ALERT_<CHANNEL>_MIN_SEVERITY | (inherit global) | Per-channel severity floor that beats the global default for one channel only (<CHANNEL> ∈ WEBHOOK, EMAIL, OPSGENIE, PAGERDUTY, TELEGRAM, NTFY, PUSHOVER, MATRIX). Example: HEALTH_ALERT_WEBHOOK_MIN_SEVERITY=warning with a critical global sends warnings to the webhook only. Values: warning or critical. Twilio keeps its own HEALTH_ALERT_TWILIO_MIN_SEVERITY. |
HEALTH_ALERT_<CHANNEL>_ENABLED | (inherit) | Set false to silence one channel entirely without unsetting its config (<CHANNEL> as above). HEALTH_ALERT_WEBHOOK_ENABLED=false disables the webhook channel — routes + the legacy global URL — distinct from the master HEALTH_ALERT_ENABLED. Per-channel flags are optional: a configured channel is on by default. |
HEALTH_ALERT_TELEGRAM_BOT_TOKEN | (unset = Telegram disabled) | Telegram Bot API token. Server-only secret — never exposed to the browser. Set together with HEALTH_ALERT_TELEGRAM_CHAT_ID to enable the global Telegram channel; each finding is sent to the chat via the Bot API sendMessage endpoint. Per-rule/per-host Telegram routes (Routing tab) override this fallback. |
HEALTH_ALERT_TELEGRAM_CHAT_ID | (unset = Telegram disabled) | Target Telegram chat id (e.g. -1001234567890, or @channelname). Both this and HEALTH_ALERT_TELEGRAM_BOT_TOKEN must be set for the global Telegram channel to fire. |
HEALTH_ALERT_MATRIX_HOMESERVER_URL | (unset = native Matrix disabled) | Matrix homeserver base URL (e.g. https://matrix.example.org). Set together with HEALTH_ALERT_MATRIX_ACCESS_TOKEN and HEALTH_ALERT_MATRIX_ROOM_ID to post each finding to a room through the Client-Server API. Per-rule/per-host Matrix routes (Routing tab) override this fallback. See Matrix / Element below. |
HEALTH_ALERT_MATRIX_ACCESS_TOKEN | (unset = native Matrix disabled) | Access token of the bot user, sent as Authorization: Bearer <token>. Server-only secret — never exposed to the browser. |
HEALTH_ALERT_MATRIX_ROOM_ID | (unset = native Matrix disabled) | Room id (!abc123:example.org), not an alias. |
HEALTH_ALERT_NTFY_URL | (unset = ntfy disabled) | Full ntfy topic URL (e.g. https://ntfy.sh/my-topic, or a self-hosted server). Each finding is published with Title/Priority/Tags headers + a plain-text body (severity → priority: critical 5/urgent, warning 4/high, recovery 3/default). Per-rule/per-host ntfy routes (Routing tab) override this fallback. |
HEALTH_ALERT_NTFY_TOKEN | (unset = no auth) | Optional ntfy access token for a protected topic, sent as Authorization: Bearer <token>. Server-only secret — never exposed to the browser. |
HEALTH_ALERT_TWILIO_ACCOUNT_SID | (unset = Twilio disabled) | Twilio Account SID. Set together with HEALTH_ALERT_TWILIO_AUTH_TOKEN, HEALTH_ALERT_TWILIO_FROM, and HEALTH_ALERT_TWILIO_TO to enable the global Twilio SMS channel. There are no per-rule/per-host SMS routes, but the credentials are editable from the Server delivery channels UI (see callout below). |
HEALTH_ALERT_TWILIO_AUTH_TOKEN | (unset = Twilio disabled) | Twilio Auth Token. Server-only secret — never exposed to the browser. Authenticates as HTTP Basic (AccountSid:AuthToken) against the Twilio Messages API. |
HEALTH_ALERT_TWILIO_FROM | (unset = Twilio disabled) | The Twilio phone number (E.164, e.g. +15557654321) messages are sent from. |
HEALTH_ALERT_TWILIO_TO | (unset = Twilio disabled) | Comma-separated list of SMS recipients (E.164, e.g. +15551234567,+15559876543). One SMS is sent per recipient per finding. |
HEALTH_ALERT_TWILIO_MIN_SEVERITY | critical | Minimum severity that triggers an SMS specifically (independent of HEALTH_ALERT_MIN_SEVERITY above). Values: warning or critical. Defaults to critical because SMS is a last-resort paging channel that costs real money per message — set to warning to also page on warnings. |
HEALTH_ALERT_PUSHOVER_TOKEN | (unset = Pushover disabled) | Pushover application API token. Server-only secret — never exposed to the browser. Set together with HEALTH_ALERT_PUSHOVER_USER to enable the global Pushover channel; each finding is POSTed to the Messages API (severity → priority: critical 2/emergency with retry+expire, warning 0/normal, recovery -1/low-quiet). Per-rule/per-host Pushover routes (Routing tab) override this fallback. |
HEALTH_ALERT_PUSHOVER_USER | (unset = Pushover disabled) | Target Pushover user or group key. Both this and HEALTH_ALERT_PUSHOVER_TOKEN must be set for the global Pushover channel to fire. |
HEALTH_ALERT_DIGEST_MINUTES | 0 (off) | Optional time-window digest for alert batching. 0/unset dispatches every finding on the sweep tick (in-pass grouping still applies — see the callout below). A positive value buffers non-critical findings and flushes them together once the window elapses, collapsing a burst into one message per target; critical findings always bypass the buffer and page immediately. Requires a D1 binding for the buffer; with none it degrades to 0 (off). Overridable from Health → Settings → Alerts → Digest batching. |
HEALTH_ALERT_INSTANCE_NAME | (unset) | Deployment label, e.g. prod-eu. Every alert and digest starts with [prod-eu]; Hookshot also uses it as the sender name. Use it when several deployments post into the same channel. |
HEALTH_ALERT_DASHBOARD_URL | (unset) | Public http(s) base URL of this dashboard. Alerts add an Open dashboard link to <url>/health?host=<id> (Slack button, Discord embed link, Matrix/Hookshot/Google Chat/Teams/Telegram link, dashboardUrl in raw JSON). The cron sweep has no request origin, so this must be set. |
HEALTH_ALERT_METADATA | (unset) | Non-secret key/values, key=value,key2=value2 or a JSON object (max 10 pairs, values ≤128 chars). Shown with the instance name in a context line, sent as metadata in raw JSON, and available to custom templates as {{meta.<key>}}. |
Built-in alert rules
The sweep evaluates these rules on every host. Thresholds are the defaults;
override them per rule from the UI, alerts.yaml thresholds:, or
HEALTH_THRESHOLD_<RULE>_WARNING / _CRITICAL. A rule whose system table is
missing (for example no system.backup_log) is skipped, not failed.
backups
- Failed Backups (24h)
failed-backupswarning ≥ 1critical ≥ 3Backup operations that ended in FAILED status in the last 24 hours.
- Requires
system.backup_log(skipped when missing)- Alert reads
- 1 failed backup
- Remote storage unavailable or credentials expired
- Disk full on backup destination
- Network interruption during backup transfer
Rule SQL
SELECT count() AS failed_count FROM system.backup_log WHERE event_time_microseconds > now() - INTERVAL 24 HOUR AND status IN ('BACKUP_FAILED', 'RESTORE_FAILED')
dictionaries
- Dictionary Load Failures
dictionary-failureswarning ≥ 1critical ≥ 5Dictionaries whose last load or reload failed. A dictionary that never loaded breaks every dictGet on it; one whose reload failed keeps serving stale data. Fix the source named in last_exception, then SYSTEM RELOAD DICTIONARY.
- Requires
system.dictionaries(skipped when missing)- Alert reads
- 1 failing dictionary
Rule SQL
SELECT countIf(last_exception != '') AS failed_count FROM system.dictionariesGet failing dictionaries
SELECT database, name, status, last_successful_update_time, substring(last_exception, 1, 300) AS last_exception FROM system.dictionaries WHERE last_exception != '' ORDER BY database, name LIMIT 20
logs
- Fatal Log Entries
fatal-log-entrieswarning ≥ 1critical ≥ 5Fatal errors in the server text log in the last hour.
- Requires
system.text_log(skipped when missing)- Alert reads
- 1 fatal log entries
Rule SQL
SELECT count() AS fatal_count FROM system.text_log WHERE level = 'Fatal' AND event_time >= now() - INTERVAL 1 HOUR
other
- Concurrent Query Saturation
concurrent-querieswarning ≥ 80critical ≥ 95Running queries as a percentage of max_concurrent_queries. At 100% every new query fails with TOO_MANY_SIMULTANEOUS_QUERIES — find what is holding slots (long or stuck queries, a client retry storm).
- Requires
system.server_settings(skipped when missing)- Alert reads
- 80% of max_concurrent_queries
Rule SQL
SELECT round( (SELECT value FROM system.metrics WHERE metric = 'Query') * 100.0 / nullIf(toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_concurrent_queries')), 0), 1 ) AS concurrency_percentGet running queries by user
SELECT user, count() AS running, round(max(elapsed)) AS oldest_seconds, any(substring(query, 1, 200)) AS sample_query FROM system.processes GROUP BY user ORDER BY running DESC LIMIT 20 - Delayed Inserts
delayed-insertswarning ≥ 1critical ≥ 5INSERTs currently throttled because a partition exceeded parts_to_delay_insert. Merges are falling behind inserts — batch inserts into fewer, larger blocks or coarsen PARTITION BY.
- Requires
system.metrics(skipped when missing)- Alert reads
- 1 delayed insert
- Too-small batches producing many parts (recommend ≥ 1000 rows / insert)
- Background merges starved (low background pool size, slow disk)
- `parts_to_delay_insert` / `parts_to_throw_insert` tuned too low
Rule SQL
SELECT value AS delayed_inserts FROM system.metrics WHERE metric = 'DelayedInserts'Get partitions with the most parts
SELECT database, table, partition, count() AS parts FROM system.parts WHERE active GROUP BY database, table, partition ORDER BY parts DESC LIMIT 20 - Memory Pressure
memory-pressurewarning ≥ 80critical ≥ 90Server memory tracked by ClickHouse (MemoryTracking) as a percentage of the server memory limit (max_server_memory_usage). Near 100% queries fail with MEMORY_LIMIT_EXCEEDED — find the heaviest queries and cap max_memory_usage.
- Requires
system.asynchronous_metrics(skipped when missing)- Alert reads
- 80% of the server memory limit
Rule SQL
SELECT round( (SELECT value FROM system.metrics WHERE metric = 'MemoryTracking') * 100.0 / nullIf(coalesce( nullIf(toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage')), 0), (SELECT value FROM system.asynchronous_metrics WHERE metric = 'OSMemoryTotal') * toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage_to_ram_ratio')) ), 0), 1 ) AS memory_percentGet top queries by memory
SELECT query_id, user, elapsed, formatReadableSize(memory_usage) AS memory, substring(query, 1, 200) AS query FROM system.processes ORDER BY memory_usage DESC LIMIT 20 - Long-Running Query
long-running-querieswarning ≥ 600critical ≥ 3600Seconds the longest running query has been running (system.processes, initial queries only), naming up to 5 as user:query_id. A monitoring signal: shown on /health and fed to "Runaway query under system pressure", never sent to a channel on its own.
- Alert reads
- longest query running 600s
- Missing PRIMARY KEY / sort order for the query pattern
- Cross-shard JOINs without distributed_product_mode tuning
- Stuck query waiting on lock or external resource
Rule SQL
SELECT round(max(elapsed)) AS longest_seconds, arrayMap(t -> concat(t.2, ':', t.3), arraySlice(arrayReverseSort(groupArray((elapsed, user, query_id))), 1, 5)) AS longest_queries FROM system.processes WHERE is_initial_query = 1 AND query_id != queryID()Get longest running queries
SELECT query_id, user, round(elapsed) AS elapsed_seconds, formatReadableSize(memory_usage) AS memory, substring(query, 1, 200) AS query FROM system.processes WHERE is_initial_query = 1 ORDER BY elapsed DESC LIMIT 20 - Heavy Query Memory
heavy-query-memorywarning ≥ 30critical ≥ 50Memory of the single heaviest running query as a percentage of the server memory limit (the same limit Memory Pressure uses), naming up to 5 as user:query_id. A monitoring signal: shown on /health and fed to "Runaway query under system pressure", never sent to a channel on its own.
- Requires
system.asynchronous_metrics(skipped when missing)- Alert reads
- heaviest query uses 30% of the server memory limit
Rule SQL
SELECT round( max(memory_usage) * 100.0 / nullIf(coalesce( nullIf(toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage')), 0), (SELECT value FROM system.asynchronous_metrics WHERE metric = 'OSMemoryTotal') * toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage_to_ram_ratio')) ), 0), 1 ) AS heaviest_query_memory_percent, arrayMap(t -> concat(t.2, ':', t.3), arraySlice(arrayReverseSort(groupArray((memory_usage, user, query_id))), 1, 5)) AS heaviest_queries FROM system.processes WHERE is_initial_query = 1 AND query_id != queryID()Get top queries by memory
SELECT query_id, user, round(elapsed) AS elapsed_seconds, formatReadableSize(memory_usage) AS memory, substring(query, 1, 200) AS query FROM system.processes WHERE is_initial_query = 1 ORDER BY memory_usage DESC LIMIT 20 - Broken Detached Parts
broken-detached-partswarning ≥ 1critical ≥ 1Parts ClickHouse detached because they were broken (checksum mismatch, corruption, unexpected files). On a replica the data is usually refetched; on a non-replicated table it may be lost — inspect before dropping.
- Requires
system.detached_parts(skipped when missing)- Alert reads
- 1 broken detached part
Rule SQL
SELECT count() AS broken_count, groupArray(5)(concat(database, '.', table, ' ', name)) AS affected FROM system.detached_parts WHERE reason LIKE 'broken%'Get broken detached parts
SELECT database, table, name, reason FROM system.detached_parts WHERE reason LIKE 'broken%' ORDER BY database, table, name LIMIT 50 - Detached Parts To Review
detached-parts-reviewwarning ≥ 1critical ≥ 1000000Off by default; enable with HEALTH_ALERT_ENABLE_RULES=detached-parts-review. Detached parts hold disk space until someone reviews them. The value is the larger of two ratios: detached bytes against 1% of total disk space, and the oldest detached part against 30 days. 1 means one limit is reached. Coverage is per replica: this host only, a part detached here says nothing about the other replicas, so check each replica. Open the Detached Parts Triage page (/detached-parts-triage) to review each part before acting.
- Requires
system.detached_parts(skipped when missing)- Alert reads
- detached parts at 1.00x the review limit (1% of disk or 30 days)
Rule SQL
SELECT if(count() = 0, 0, greatest( ifNull(sum(bytes_on_disk) * 100.0 / nullIf((SELECT sum(total_space) FROM system.disks), 0), 0), ifNull((toUnixTimestamp(now()) - toUnixTimestamp(min(modification_time))) / 86400.0 / 30, 0) )) AS review_ratio FROM system.detached_partsList detached parts to review
SELECT database, table, name, reason, bytes_on_disk, modification_time FROM system.detached_parts ORDER BY modification_time ASC, bytes_on_disk DESC LIMIT 50
parts & merges
- Failed Mutations
failed-mutationswarning ≥ 1critical ≥ 5Mutations that are not complete and have recorded a failure. Failed mutations block subsequent mutations on the same table.
- Requires
system.mutations(skipped when missing)- Alert reads
- 1 failed mutation
- Mutation references a column that no longer exists
- Disk full on a replica preventing part writes
- Hardware / network fault during part rewrite
Rule SQL
SELECT countIf(is_done = 0 AND latest_fail_reason != '') AS failed_count, groupArrayIf(5)(concat(database, '.', table, ' ', mutation_id), is_done = 0 AND latest_fail_reason != '') AS affected FROM system.mutationsGet failed mutations
SELECT database, table, mutation_id, command, latest_fail_reason, latest_fail_time FROM system.mutations WHERE is_done = 0 AND latest_fail_reason != '' ORDER BY latest_fail_time DESC LIMIT 20 - Stuck Merges
stuck-mergeswarning ≥ 1critical ≥ 3Merges running for more than 10 minutes. Stuck merges block table inserts and consume resources.
- Requires
system.merges(skipped when missing)- Alert reads
- 1 stuck merge
- Very large parts requiring many hours to merge
- Disk I/O saturation or slow storage
- Insufficient background_pool_size
Rule SQL
SELECT count() AS stuck_count, groupArray(5)(concat(database, '.', table, ' ', result_part_name)) AS affected FROM system.merges WHERE elapsed > 600Get stuck merges
SELECT database, table, elapsed, progress, num_parts, total_size_bytes_compressed FROM system.merges WHERE elapsed > 600 ORDER BY elapsed DESC LIMIT 20 - Parts Pressure
parts-pressurewarning ≥ 80critical ≥ 95Worst partition’s active parts as a percentage of parts_to_throw_insert. A predictive "too many parts" signal — sustained pressure leads to throttled then rejected inserts.
- Requires
system.parts(skipped when missing)- Alert reads
- 80% of parts_to_throw_insert
- Frequent small inserts producing parts faster than merges consume them
- Background merges starved (low background_pool_size, slow disk)
- Partition key too granular, spreading inserts across many partitions
- `parts_to_throw_insert` / `parts_to_delay_insert` lowered per table
Rule SQL
WITH parts_now AS ( SELECT database, table, partition_id, partition, count() AS parts FROM system.parts WHERE active AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema') GROUP BY database, table, partition_id, partition ), table_overrides AS ( SELECT database, name AS table, toInt64OrZero(extract(replaceRegexpOne( replaceRegexpAll(create_table_query, '\\x27(?:[^\\x27\\\\]|\\\\.)*\\x27', ''), '(?s)^.*\\sSETTINGS\\s+', '' ), '(?:^|,)\\s*parts_to_throw_insert\\s*=\\s*([0-9]+)')) AS throw_override, toInt64OrZero(extract(replaceRegexpOne( replaceRegexpAll(create_table_query, '\\x27(?:[^\\x27\\\\]|\\\\.)*\\x27', ''), '(?s)^.*\\sSETTINGS\\s+', '' ), '(?:^|,)\\s*parts_to_delay_insert\\s*=\\s*([0-9]+)')) AS delay_override FROM system.tables WHERE database IN (SELECT name FROM system.databases WHERE engine IN ('Atomic', 'Lazy', 'Memory', 'Ordinary', 'Replicated', 'Shared')) AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema') AND (database, name) IN (SELECT database, table FROM parts_now) ), server_defaults AS ( SELECT coalesce((SELECT toInt64OrNull(value) FROM system.merge_tree_settings WHERE name = 'parts_to_throw_insert'), 3000) AS throw_default, coalesce((SELECT toInt64OrNull(value) FROM system.merge_tree_settings WHERE name = 'parts_to_delay_insert'), 1000) AS delay_default ), limits AS ( SELECT p.database AS database, p.table AS table, p.partition_id AS partition_id, p.partition AS partition, p.parts AS parts, if(o.throw_override > 0, o.throw_override, (SELECT throw_default FROM server_defaults)) AS throw_limit, if(o.delay_override > 0, o.delay_override, (SELECT delay_default FROM server_defaults)) AS delay_limit FROM parts_now p LEFT JOIN table_overrides o ON p.database = o.database AND p.table = o.table WHERE p.parts >= 20 ) SELECT round(max(parts * 100.0 / nullIf(throw_limit, 0)), 1) AS pressure_percent FROM limitsGet partitions near the throw limit
SELECT database, table, partition, count() AS parts FROM system.parts WHERE active GROUP BY database, table, partition ORDER BY parts DESC LIMIT 20 - TTL & Partition Health
ttl-partition-healthwarning ≥ 1critical ≥ 5MergeTree tables with partition bloat, a time-based PARTITION BY and no table TTL, or a merge backlog. Recommend-only — never applies ALTER TTL or DROP PARTITION.
- Requires
system.parts(skipped when missing)- Alert reads
- 1 table to review
- PARTITION BY too granular (hourly or high-cardinality keys)
- Time-series table with no table-level TTL, so old partitions never drop
- Inserts producing parts faster than background merges consume them
Rule SQL
WITH mergetree_tables AS ( SELECT database, name, engine, partition_key, engine_full FROM system.tables WHERE is_temporary = 0 AND database IN (SELECT name FROM system.databases WHERE engine IN ('Atomic', 'Lazy', 'Memory', 'Ordinary', 'Replicated', 'Shared')) AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema') AND positionCaseInsensitive(engine, 'MergeTree') > 0 ), part_stats AS ( SELECT database, table, uniqExact(partition) AS partitions, count() AS active_parts, if( uniqExact(partition) = 0, 0, round(count() / uniqExact(partition), 2) ) AS parts_per_partition, sum(bytes_on_disk) AS bytes_on_disk, sum(rows) AS rows FROM system.parts WHERE active AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema') GROUP BY database, table ), inventory AS ( SELECT t.database, t.name AS table, concat(t.database, '.', t.name) AS full_table, t.database AS _database, t.name AS _table, t.engine, t.partition_key, if( positionCaseInsensitive(t.engine_full, ' TTL ') > 0, trim(BOTH ' ' FROM replaceRegexpOne( substring( t.engine_full, positionCaseInsensitive(t.engine_full, ' TTL ') + 5 ), '\\s+SETTINGS\\s+.*$', '' )), '' ) AS ttl_expression, multiIf( ifNull(p.partitions, 0) >= 1000, 'Rebuild with coarser PARTITION BY', ifNull(p.partitions, 0) >= 500, 'Consider monthly partitions', ( positionCaseInsensitive(t.partition_key, 'toYYYYMM') > 0 OR positionCaseInsensitive(t.partition_key, 'toStartOf') > 0 OR positionCaseInsensitive(t.partition_key, 'toDate') > 0 OR positionCaseInsensitive(t.partition_key, 'toMonday') > 0 OR positionCaseInsensitive(t.partition_key, 'toYearWeek') > 0 OR positionCaseInsensitive(t.partition_key, 'toISOWeek') > 0 ) AND positionCaseInsensitive(t.engine_full, ' TTL ') = 0, 'Add table TTL', ifNull(p.partitions, 0) > 0 AND ifNull(p.active_parts, 0) / ifNull(p.partitions, 1) >= 10, 'Check merge backlog', '' ) AS recommendation, ifNull(p.partitions, 0) AS partitions, ifNull(p.active_parts, 0) AS active_parts, ifNull(p.parts_per_partition, 0) AS parts_per_partition, ifNull(p.bytes_on_disk, 0) AS bytes_on_disk, ifNull(p.rows, 0) AS rows FROM mergetree_tables AS t LEFT JOIN part_stats AS p ON p.database = t.database AND p.table = t.name ) SELECT count() AS flagged_count FROM inventory WHERE recommendation != ''Get tables with 500+ partitions
SELECT concat(database, '.', table) AS full_table, uniqExact(partition) AS partitions, count() AS active_parts FROM system.parts WHERE active GROUP BY database, table HAVING partitions >= 500 ORDER BY partitions DESC LIMIT 20
queries
- Query Timeout Breaches (1h)
query-timeoutwarning ≥ 1critical ≥ 10Queries killed due to timeout (TIMEOUT_EXCEEDED) in the last hour.
- Requires
system.query_log(skipped when missing)- Alert reads
- 1 timeout kills in last hour
- max_execution_time set too low for heavy analytical queries
- Missing sort key for the filter predicate
- Cross-shard queries without proper distribution settings
Rule SQL
SELECT count() AS timeout_count, groupArray(5)(query_id) AS affected FROM system.query_log WHERE event_time > now() - INTERVAL 1 HOUR AND type IN ('ExceptionWhileProcessing', 'ExceptionBeforeStart') AND (exception_code = 159 OR exception LIKE '%TIMEOUT_EXCEEDED%') - Rejected Inserts (1h)
rejected-insertswarning ≥ 1critical ≥ 1INSERTs rejected with TOO_MANY_PARTS in the last hour. Data was not written — the client must retry. Reduce insert frequency or coarsen PARTITION BY.
- Requires
system.query_log(skipped when missing)- Alert reads
- 1 rejected inserts in last hour
Rule SQL
SELECT count() AS rejected_count, groupArray(5)(query_id) AS affected FROM system.query_log WHERE event_time > now() - INTERVAL 1 HOUR AND type IN ('ExceptionBeforeStart', 'ExceptionWhileProcessing') AND exception_code = 252
replication
- Readonly Replicas
readonly-replicaswarning ≥ 1critical ≥ 3Replicas in read-only mode cannot accept writes.
- Requires
system.replicas(skipped when missing)- Alert reads
- 1 readonly replica
- ZooKeeper / ClickHouse Keeper unavailable or laggy
- Replica metadata corruption (force_restore_data required)
- Disk full on the replica
Rule SQL
SELECT count() AS readonly_count, groupArray(5)(concat(database, '.', table)) AS affected FROM system.replicas WHERE is_readonly = 1 - Replication Lag
replication-lagwarning ≥ 30critical ≥ 300Maximum absolute_delay across all replicas (seconds behind leader).
- Requires
system.replicas(skipped when missing)- Alert reads
- 30s max delay
- Slow network or saturated inter-server connections
- Large fetch backlog in `replication_queue`
- ZooKeeper / Keeper latency
Rule SQL
SELECT max(absolute_delay) AS max_lag FROM system.replicasGet lagging replicas
SELECT database, table, replica_name, absolute_delay, is_readonly FROM system.replicas ORDER BY absolute_delay DESC LIMIT 20 - Stuck Replication Queue
replication-queue-stuckwarning ≥ 1critical ≥ 10Replication queue entries retried more than 100 times or queued for over an hour. A stuck entry stops the replica from catching up and grows replication lag.
- Requires
system.replication_queue(skipped when missing)- Alert reads
- 1 stuck queue entry
Rule SQL
SELECT count() AS stuck_count, groupArray(5)(concat(database, '.', table, ' ', node_name)) AS affected FROM system.replication_queue WHERE num_tries > 100 OR create_time < now() - INTERVAL 1 HOURGet stuck queue entries
SELECT database, table, type, create_time, num_tries, last_exception, postpone_reason FROM system.replication_queue WHERE num_tries > 100 OR create_time < now() - INTERVAL 1 HOUR ORDER BY create_time ASC LIMIT 20 - Replica Keeper Session Expired
replica-session-expiredwarning ≥ 1critical ≥ 3Replicated tables whose ZooKeeper/Keeper session has expired. These tables are read-only until the session is re-established — check Keeper health and network to the Keeper ensemble.
- Requires
system.replicas(skipped when missing)- Alert reads
- 1 expired replica session
Rule SQL
SELECT countIf(is_session_expired) AS expired_count, groupArrayIf(5)(concat(database, '.', table), is_session_expired) AS affected FROM system.replicasGet replicas with an expired session
SELECT database, table, replica_name, is_readonly, zookeeper_path FROM system.replicas WHERE is_session_expired LIMIT 20 - Stuck Distributed Send Queue
distribution-queuewarning ≥ 1critical ≥ 5Distributed tables whose async insert queue is not draining: queued files with send errors or a blocked queue, or broken files that were never sent. The data is not on the shards yet, and the queue grows on local disk — check the shard named in last_exception.
- Requires
system.distribution_queue(skipped when missing)- Alert reads
- 1 Distributed table with a stuck queue
Rule SQL
SELECT countIf((data_files > 0 AND (error_count > 0 OR is_blocked)) OR broken_data_files > 0) AS stuck_tables FROM system.distribution_queueGet stuck Distributed queues
SELECT database, table, is_blocked, error_count, data_files, formatReadableSize(data_compressed_bytes) AS queued, broken_data_files, last_exception_time, substring(last_exception, 1, 300) AS last_exception FROM system.distribution_queue WHERE data_files > 0 OR broken_data_files > 0 ORDER BY error_count DESC, data_files DESC LIMIT 20
storage
- Disk Usage
disk-usagewarning ≥ 80critical ≥ 95Worst-case disk utilization across all configured volumes.
- Requires
system.disks(skipped when missing)- Alert reads
- 80% used (worst disk)
Rule SQL
SELECT round(max((total_space - free_space) * 100.0 / nullIf(total_space, 0)), 1) AS disk_percent FROM system.disks - Disk Fill Forecast
disk-time-to-fullwarning ≥ 100critical ≥ 400Fits a linear trend to each disk’s free space over the last 6 hours and reports how much of the current free space that trend uses up in 24 hours. 100% means the disk fills within a day, 400% within 6 hours. Catches a fast-filling disk long before disk-usage crosses its percent threshold.
- Requires
system.asynchronous_metric_log(skipped when missing)- Alert reads
- full in ~24h at the current rate
Rule SQL
SELECT round(maxOrNull(fill_pct), 1) AS fill_24h_percent FROM ( SELECT argMax(value, event_time) AS free_now, simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 AS bytes_per_second, if(free_now <= 0, 10000, least(greatest(-bytes_per_second, 0) * 86400 * 100 / free_now, 10000)) AS fill_pct FROM system.asynchronous_metric_log WHERE startsWith(metric, 'DiskAvailable_') AND event_date >= toDate(now() - INTERVAL 6 HOUR) AND event_time > now() - INTERVAL 6 HOUR GROUP BY metric HAVING max(event_time) - min(event_time) >= 3 * 3600 AND count() >= 30 )Get disk fill rates
SELECT replaceOne(metric, 'DiskAvailable_', '') AS disk, formatReadableSize(argMax(value, event_time)) AS free_now, formatReadableSize(-simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 * 3600) AS used_per_hour, if(simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 < 0, round(argMax(value, event_time) / -simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 / 3600, 1), NULL) AS hours_to_full FROM system.asynchronous_metric_log WHERE startsWith(metric, 'DiskAvailable_') AND event_date >= toDate(now() - INTERVAL 6 HOUR) AND event_time > now() - INTERVAL 6 HOUR GROUP BY metric ORDER BY hours_to_full ASC NULLS LAST LIMIT 20
views
- MV Refresh Failures
mv-refresh-failureswarning ≥ 1critical ≥ 3Materialized views with REFRESH schedule that have failed or errored their last refresh cycle.
- Requires
system.view_refreshes(skipped when missing)- Alert reads
- 1 failed MV refresh
- Source table schema changed incompatibly
- MV query references a missing table or column
- Resource exhaustion during refresh
Rule SQL
SELECT countIf(exception != '' OR retry > 0) AS failed_count, groupArrayIf(5)(concat(database, '.', view), exception != '' OR retry > 0) AS affected FROM system.view_refreshes
compound
- Replica Split-Brain Risk
replica-split-brainReplication lag AND readonly replicas both firing at once — a stronger signal of a stuck/diverging replica than either metric alone.
Fires when all of:
replication-lag,readonly-replicas - Merge Pressure
merge-pressureStuck merges AND high disk usage both firing at once — merges are likely stalled fighting for disk headroom rather than transient load.
Fires when all of:
stuck-merges,disk-usage - Runaway Query Under System Pressure
runaway-query-impactA long-running or memory-heavy query (at least warning) while the server is critical on memory or concurrent query slots. The only page for heavy queries: a heavy query on a healthy server stays a signal on /health.
Fires when all of:
long-running-queries,heavy-query-memory,memory-pressure,concurrent-queries
Rules that report a count also name what they count, so the alert says which
tables, parts, mutations, views or queries to look at. The label lists up to
5 names, then "and N more", for example 7 stuck merges: db.events all_1_9_2, db.logs all_3_7_1 and 5 more. The names come from the same query
(groupArray(5)), so they add no extra statement. Each name is cut to 80
characters and escaped in HTML, Slack and Teams bodies.
keeper-unavailable warns at 5, not 1: one KEEPER_EXCEPTION an hour is
routine (a session reconnect or leader re-election). Sustained exceptions are
the quorum signal. rejected-inserts reads a one-hour query_log window
because the RejectedInserts event in system.events is cumulative since
server start and would keep firing after the incident ended. A rejected insert
or a broken part is possible data loss, so both go straight to critical.
memory-pressure divides tracked memory by max_server_memory_usage, the
limit at which queries fail, not by host RAM. In a container that limit
follows the cgroup memory limit.
distribution-queue counts Distributed tables whose async insert queue has
send errors, is blocked, or holds broken files. A queue that drains normally
never counts, so a shard outage is the usual cause: the data waits on the
sender's disk until the shard is back.
concurrent-queries compares running queries with max_concurrent_queries
and is skipped when the limit is 0 (unlimited). At 100% the sweep's own
query is refused too, so a saturated host shows a failed check rather than
this alert; the 80% and 95% thresholds are the warning before that.
dictionary-failures counts dictionaries with a last_exception. That
includes a reload that failed on a dictionary still marked LOADED, which
keeps answering dictGet with stale data. A dictionary that was never loaded
(lazy loading) has no exception and does not count.
disk-time-to-full fits a straight line to each disk's free space over the
last 6 hours of system.asynchronous_metric_log and reports how much of the
free space that trend uses in 24 hours: 100 means full within a day
(warning), 400 within 6 hours (critical). A disk that is not shrinking reads
0. It needs asynchronous_metric_log enabled and at least 3 hours of
samples per disk; until then it reports nothing. disk-usage still alerts on percent
used.
long-running-queries reports how long the longest running query has been
running (warning 10 minutes, critical 1 hour). heavy-query-memory reports the
memory of the heaviest running query as a percentage of the same server memory
limit memory-pressure uses (warning 30%, critical 50%). Both name up to 5
queries as user:query_id, longest or heaviest first. Both are signals, not
pages: they show on the Health page but never send an alert on their own. A
long report query on a healthy server is normal and should not wake anyone.
runaway-query-impact is the page for heavy queries. It fires, always as
critical, only when a long or heavy query (at least warning) runs while the
server itself is critical: memory-pressure or concurrent-queries is
critical. Put simply, a heavy query alone is shown, and it pages only when the
whole system is in trouble at the same time. The alert names the offending
queries so you can kill the right one. To also get alerts for one of the two
signals on its own, list it in HEALTH_ALERT_NOTIFY_RULES.
Alert grouping & digest batching
When one sweep finds several alerts bound for the same delivery target (e.g. disk filling on 8 hosts, all routed to one Slack channel), they are combined into one message — a summary line (3 critical, 2 warning on 4 hosts) followed by one row per rule and severity — instead of N separate pings. Findings of the same rule share a row with the rule title, a count and up to three example names ([WARNING] Source stopped sending rows · 34 (source-1, source-2, source-3, +31)); a rule with one finding keeps its full line. When several findings share one host the header names it (on 1 host (node-1)) and the rows leave it out. A sweep that only adds findings of a rule that is already open, or only resolves, sends one short line to Slack and generic webhook targets (+1 warning · 1 resolved · 60 open) instead of the whole table; the first finding of a rule, an escalation and a reminder still send their normal message. Matrix and Hookshot already list findings under a heading per rule, and the generic webhook still carries every finding in alerts. A reminder for a condition that is still firing takes one short line (still open, 3 h (82.8 -> 82.9), with how the value moved since the last notification when both values are numbers) instead of the full detail, also when it is the only finding. A Slack message with ack buttons keeps its card. This in-pass grouping is always on and needs no configuration; it applies to the generic webhook, Slack, and Telegram channels, to native Matrix rooms, and to Matrix Hookshot and Element / Matrix custom webhook targets that have no title or body template (every other channel keeps per-finding sends). Native Matrix rooms and Hookshot / Matrix targets are always grouped within the sweep: the time-window digest never buffers them, because they carry secrets (an access token or headers). To also batch across time, enable the optional time-window digest (HEALTH_ALERT_DIGEST_MINUTES or the Alerts tab): non-critical findings wait up to the window and flush together, while criticals still page immediately.
Edit channels from the UI (no redeploy)
Every channel above is also editable from Health → Settings → Alerts → Server delivery channels. A channel saved there is persisted per-owner (in D1, when configured) and takes precedence over its HEALTH_ALERT_* environment variable on the next sweep — so you can add or rotate a destination without a redeploy. Leaving a secret field blank on save keeps the stored secret. A channel with no saved config falls back to its env variable (shown as "Configured via server env"), and a deployment with no D1 binding simply keeps using the env variables. Browser notifications stay per-browser (local to your session).
SMS costs real money
Every Twilio dispatch is a billed SMS — one message per configured recipient, per finding. HEALTH_ALERT_TWILIO_MIN_SEVERITY defaults to critical so routine warnings never page a phone; only raise it to warning if you understand the cost. Message bodies are also truncated to Twilio's 1600-character SMS limit.
CRON_SECRET is required
CRON_SECRET is required for the cron endpoints. When it is unset, /api/cron/health-sweep and /api/cron/retention-prune fail closed and return HTTP 503 (they do not run) — retention-prune is destructive, so it must never be reachable unauthenticated. Set CRON_SECRET and pass it as Authorization: Bearer <secret> from your cron caller.
Set secrets
Example for Cloudflare Workers (using wrangler secret put):
wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URL
# Optional: Telegram channel (bot token is a secret; chat id is a plain var)
wrangler secret put HEALTH_ALERT_TELEGRAM_BOT_TOKEN
# Optional: ntfy channel (topic URL is a plain var; access token is a secret)
wrangler secret put HEALTH_ALERT_NTFY_TOKEN
# Optional: Twilio SMS channel (auth token is a secret; SID/from/to are plain vars)
wrangler secret put HEALTH_ALERT_TWILIO_AUTH_TOKEN
# Optional: Pushover channel (app token is a secret; user key is a plain var)
wrangler secret put HEALTH_ALERT_PUSHOVER_TOKEN
# Optional: native Matrix (access token is a secret; homeserver URL and room id are plain vars)
wrangler secret put HEALTH_ALERT_MATRIX_ACCESS_TOKENEnable alerting
Example environment block:
HEALTH_ALERT_MIN_SEVERITY=warning
HEALTH_ALERT_WEBHOOK_URL=https://hooks.slack.com/services/...
CRON_SECRET=<random-secret>Call it manually
curl -H "Authorization: Bearer $CRON_SECRET" \
https://your-chmonitor.example.com/api/cron/health-sweepThe endpoint returns a JSON array of check results. It always returns HTTP 200; alert dispatch happens server-side.
Matrix / Element
chmonitor can post alerts to a Matrix room itself, through the Matrix
Client-Server API. It sends each alert as an m.notice event with
PUT /_matrix/client/v3/rooms/{roomId}/send/m.room.message/{txnId}. No bridge
is needed.
- Create a bot user on your homeserver, for example
@chmonitor:example.org. - Get its access token. Sign in as the bot in Element and copy the token from
Settings → Help & About → Access token, or call
POST /_matrix/client/v3/login. - Invite the bot to the room, then sign in as the bot and accept the invite. The bot must already be joined: chmonitor does not join rooms.
- Copy the room id (
!abc123:example.org) from Room settings → Advanced in Element. A room alias (#ops:example.org) does not work. - Set
HEALTH_ALERT_MATRIX_HOMESERVER_URL,HEALTH_ALERT_MATRIX_ACCESS_TOKENandHEALTH_ALERT_MATRIX_ROOM_ID, or add a Matrix / Element room route on the Routing tab and click Send test message.
A homeserver URL saved from the Routing tab, or set in a routing.yaml target,
must be public HTTPS with no username, password, query or fragment; private and
internal addresses are refused, both when you save it and again at every send.
It is stored without trailing slashes (a path prefix such as /matrix is kept).
The env URL is set by the operator and may be http and point at a LAN
homeserver; an env URL or room id that is not valid disables native Matrix and
logs one warning. Redirects are never followed, so the access token
only goes to the host you configured.
Encrypted rooms
chmonitor sends events unencrypted. In an end-to-end encrypted room the alerts still arrive, but they are not encrypted. Use a room without encryption for alerts, or accept that alert text is readable by the homeserver.
Native Matrix or Hookshot? A Hookshot (or other bridge) webhook is a custom
webhook target with the hookshot or matrix format: chmonitor POSTs to the
bridge URL and the bridge posts into the room, and findings from one sweep are
grouped into one message. Native Matrix needs no bridge and groups the same
way: each room gets one message per sweep (a lone alert keeps the full
single-alert message). Neither is held by the time-window digest, because both
carry a secret (headers or an access token) that is never written to the
buffer. If a grouped send fails, its alerts are retried on the next sweep.
A custom webhook target with the slack format is grouped the same way: one
sweep sends one Slack message with a summary line, the findings listed (capped
at 25 with "…and N more", and kept inside Slack's text limits), a colour bar
from the highest severity (green when every finding is a recovery), and an
"Open dashboard" button. A lone finding keeps the full single-alert message. A
target with a title or body template still sends one message per finding.
Scheduling (Cloudflare Cron)
The hosted deploy schedules the sweep every 10 minutes. In wrangler.toml:
[triggers]
crons = ["0 3 * * *", "0 8 * * 1", "*/10 * * * *"] # retention, weekly-report, health-sweepThe Cloudflare scheduled trigger is routed to the route's GET handler. Two gates apply, in order:
- Auth — the handler requires
CRON_SECRET(returns HTTP 503 when unset). Configure the secret and invoke withAuthorization: Bearer <secret>only — e.g. from an external scheduler, or a Worker Cron that forwards the secret. Query-string?secret=is rejected. - Enablement — after auth, the handler consults
CHM_HEALTH_SWEEP_ENABLED. When it is falsy the route returns200 {"skipped": true}without running the sweep, so you can pause scheduled alerting without touching the cron or the secret. When unset it defaults to enabled (sinceCRON_SECRETis already present at that point).
The */10 cadence (rather than */5) leaves CPU-time headroom: the sweep also generates AI insights per host, so its cost scales with the number of configured CLICKHOUSE_HOST entries. For very large host fleets, either lengthen the cadence or pause the scheduled sweep (CHM_HEALTH_SWEEP_ENABLED=false) and drive it from an external scheduler.
Weekly health report
Beyond the health-sweep, a second cron job builds a proactive weekly HTML report per opted-in host: a 7-day narrative combining the AI insights engine's top findings, the number of adaptive statistical baselines fitted to that cluster, and a disk-capacity outlook from the capacity forecaster.
Each report also includes real cluster data over the window, when the host's system tables allow it (each section is fail-open and simply omitted otherwise):
- Query activity — total and failed query counts, p50/p95 duration, and a per-day sparkline (from
system.query_log). - Ingestion — rows and bytes written by INSERTs per day, with sparklines (from
system.query_log; uncompressed measurement, same model as the Traffic page). - Storage — total on-disk size plus the top 5 tables by size, each annotated with the bytes written during the window (from active
system.parts).
Reports are opt-in, never opt-out — set CHM_WEEKLY_REPORT_HOSTS to a comma-separated list of host indices (e.g. 0,2) to enable it for those hosts. Leave it unset and no reports are generated, so a self-hosted deployment stays quiet by default.
| Variable | Default | Description |
|---|---|---|
CHM_WEEKLY_REPORT_HOSTS | unset (no reports) | Comma-separated host indices to opt in, e.g. 0,2. Garbage entries are dropped; an opted-in index with no matching configured host is skipped with a warning. |
The report is generated by GET /api/cron/weekly-report, guarded by the same CRON_SECRET as the health-sweep endpoint above, and scheduled weekly on Mondays at 08:00 UTC:
[triggers]
crons = ["0 8 * * 1"]The report is always persisted, even when no delivery channel is configured — so a deployment with no webhook still accrues viewable reports. When HEALTH_ALERT_WEBHOOK_URL is set, the report is also best-effort posted to that same webhook.
View or download the most recent persisted report:
# Self-contained HTML document
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0"
# Parsed JSON summary instead
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=json"
# A specific past week (YYYY-MM-DD week-start)
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&week=2026-07-06"
# PDF (Cloudflare Browser Rendering; falls back to HTML when unavailable)
curl -L "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=pdf" -o report.pdfScheduled reports (per-user)
Beyond the env-var opt-in above, Scheduled Reports (/report-settings in the dashboard) lets each user subscribe to a weekly or monthly cluster health report delivered through their already-configured alert channels — email (full HTML), webhook, Telegram, ntfy, and Pushover. Paging channels (Opsgenie, PagerDuty, Twilio) are deliberately excluded: a digest must never page anyone.
- Cadence — monthly (1st of the month, 30-day window,
GET /api/cron/monthly-report, cron0 8 1 * *) or weekly (Mondays, 7-day window, shares the weekly cron above). On chmonitor Cloud, weekly cadence requires a paid plan; monthly is included on Free. - Generate now — build a fresh report on demand and open/download the self-contained HTML. Works on any deployment with zero delivery configuration (
POST /api/v1/reports/generate). - Download PDF — render the same report to PDF via Cloudflare Browser Rendering (
POST /api/v1/reports/generatewith{ "format": "pdf" }, or?format=pdfon the read API). PDF export is a Pro+ (data_export) capability on chmonitor Cloud. It is optional and fail-closed: when noBROWSERbinding is configured (self-hosted Docker / K8s, or any Cloudflare account without Browser Rendering) or a render fails, the request degrades gracefully to HTML (signalled by anX-Report-PDF: unavailableheader). On Cloud, a Pro+ subscriber's scheduled email delivery also gets the PDF attached automatically. - Send test report — deliver one immediately through your configured channels and see the per-channel outcome (
POST /api/v1/reports/test-send). - Audit — the subscription stores the last delivery time and a compact per-channel status.
- Fleet report — a subscription covering more than one host is delivered as ONE combined fleet report instead of N separate ones: a comparison table (findings, critical, queries, ingested bytes, disk %) followed by each host's full sections. Per-host reports are still persisted individually.
Subscriptions live in the report_subscriptions D1 table (owner-scoped, like alert channels); deployments without D1 keep the env-var webhook flow and the on-demand Generate button. The AI agent can also produce the same report conversationally via its generate_cluster_report tool.
Notes & limitations
system.error_logis checked separately by the Errors page (under Operations). The Health page usessystem.errors(in-memory error counts), which resets on server restart.- If a system table is missing (e.g., no
system.replicason a standalone node), that check is skipped with an "unavailable" state rather than a false positive. - The webhook payload is a plain Slack/Discord-compatible JSON object (
{"text": "...", "content": "..."}). Custom payload shapes are not supported in v1. - The health-sweep endpoint queries all configured
CLICKHOUSE_HOSTentries. High host counts increase sweep latency. - If PeerDB is configured (
PEERDB_API_URL), the sweep also checks every PeerDB mirror and alerts aspeerdb:<mirror>through the same channels and routes. SetCHM_FEATURE_PEERDB_ENABLED=falseto skip it. See PeerDB Monitoring.
Related
Metrics
Live counters, async metrics, and CPU profiling.
Tables
TTL & Partitions inventory and storage pages.
DBA workflows
TTL inventory, schema compare, and recommend-only advisor.
Cluster
Topology and Keeper/ZooKeeper coordination.
Feature permissions
Gate or disable features per section.
Authentication
Configure sign-in and access control.