chmonitorchmonitor

Guide

Health

At-a-glance cluster health dashboard with automated headless health-sweep alerts via webhook and Cloudflare Cron.

21 min read

Get a single status grid of cluster health across replication, merges, errors, disks, load, and parts — plus a headless sweep endpoint that fires webhook alerts on a schedule.

Prop

Type

What it does

The Health page aggregates checks across multiple system tables into a single status grid. Each check has a severity level (ok / warning / critical) and a human-readable summary. Operators use this page as a starting point for incident triage.

Checks cover:

  • Replication lag — replicas falling behind, readonly tables
  • Merge backlog — active merge count and slow merges
  • Error rate — recent errors from system.errors
  • Disk usage — free space across all disks
  • Query load — running query count and memory pressure
  • Part health — excessive part counts
  • TTL & Partition Health — MergeTree tables with partition bloat, a time-based PARTITION BY and no table TTL, or a merge backlog (recommend-only; full inventory on /ttl-partition-health)

In addition to the UI, chmonitor exposes a headless health-sweep endpoint (GET /api/cron/health-sweep) that runs the same checks and dispatches webhook alerts. This is designed to be called on a schedule (e.g., Cloudflare Cron every 5 minutes) without a browser.

Pages

PageRouteWhat it showsSystem tables
Health/healthStatus grid with per-check severity and detailssystem.metrics, system.asynchronous_metrics, system.replicas, system.merges, system.errors, system.disks, system.replication_queue, system.processes, system.query_log, system.parts, system.tables

Using it

Open /health for the status grid — each check reports ok / warning / critical with a summary. Start incident triage here, then drill into the linked feature sections for detail.

To run the same checks without a browser, call the health-sweep endpoint (see below) on a schedule.

Alert Settings layout

/alert-settings has four tabs: Alerts, Thresholds, Activity, and Advanced. The Alerts tab answers "what alerts do I have, and where do they go?" on one screen:

  1. Built-in alerts — one row per health check, with its name and stable id.
  2. More ways to alert — Suggested alerts (thresholds derived from how this cluster behaved) and Custom rules (your own rule on any metric).
  3. Channels — browser, server delivery channels, and custom webhook targets.
  4. Delivery — Routing rules (which alerts go to which channel), Webhook subscriptions (POST alert events to your own endpoints), and Digest (batch non-critical alerts).
  5. Minimum severity — the default floor every channel inherits.

The Advanced tab now holds only Silencing: Quiet hours and Maintenance windows. Each item still opens in its own dialog. Old links keep working: ?tab=routing, ?tab=webhooks, ?tab=digest, ?tab=suggested, and ?tab=custom-rules open that dialog on the Alerts tab; ?tab=maintenance and ?tab=quiet-hours open it on Advanced. ?tab=define, ?tab=delivery, and ?tab=silencing open the tab that holds that group.

On the hosted cloud dashboard, quiet hours and maintenance windows are saved for your own connections, but scheduled alert checks do not cover your own connections yet. Until they do, these windows do not silence any alert, and both panels say so.

Naming built-in alerts

Each built-in alert can be renamed from the list (1–80 characters). The name is display-only in the dashboard: notifications still use the built-in check title, and the alert keeps its check id, so a rename never resets alert state or acknowledgements. Names are stored in the metadata database; with no database the list still renders, but rename is read-only.

Permissions & access

CHM_FEATURE_HEALTH_ACCESS=authenticated
CHM_FEATURE_HEALTH_ENABLED=false
[features.health]
enabled = true
access = "authenticated"

Configuration

Run alerts without a metadata database

Alerting works on Docker and Kubernetes with no D1 and no Postgres, but there is one boundary to learn up front: declarative config can fire alerts. It cannot acknowledge them.

DataDeclarative?Why
Alert definitions: rules, thresholds, routing, channels, webhook targets, quiet hours, maintenance windows, digest settingsYesDesired state. An operator declares intent.
Alert state: firing or cleared, hysteresis streak, incident start, last-notifiedNoObservation, not intent. It cannot be authored.
ACKs, suggestion dismissals, digest bufferNoPer-user runtime state.

With no database, alert state lives in memory in each worker instance. So hysteresis streaks and incident timers reset on every restart or deploy, and Acknowledge, Clear and dismiss are disabled in the UI. POST and DELETE on /api/v1/health/ack return 501. The ClickHouse state backend (CHM_STATE_CLICKHOUSE_*) does not back the alert stores; the alert surface runs on D1 or Postgres (DATABASE_URL / POSTGRES_URL). Attach one of them if you need ACKs. See GET /api/v1/config capabilities.health for what a deployment resolved.

In-memory alert state is not migrated when you attach a database later. On the first sweep that reaches the new database, the in-memory state is discarded and alerting starts fresh from the database: a pending hysteresis streak restarts, and a condition that is still firing opens a new incident and can notify again.

Where definitions come from

Definitions merge from four sources, highest first:

  1. D1 or Postgres rows, written from the UI.
  2. Config files in CHM_HEALTH_CONFIG_DIRECTORY (default /etc/chmonitor/health.d).
  3. Environment variables (HEALTH_ALERT_*, HEALTH_THRESHOLD_*).
  4. Built-in defaults.

Every source is always read; precedence only decides who wins a key. Per key:

  • Most definitions (routes, custom rules, webhook targets, quiet hours, channels) merge field by field. A higher source overrides a field only when it sets a non-empty value, so a file can set enabled while a D1 row owns a token. A higher source cannot clear a lower source's value back to empty.
  • Maintenance windows are replaced whole per id, and windows from different sources are all active.
  • Digest settings are a single value: the highest source wins outright.
  • A key that only a lower source defines disappears when that source stops defining it. There is no tombstone.

Declared entries are shared by every user (deployment-wide, like the env fallbacks). In the UI they are read-only and show a Config file or Env badge instead of Delete or Reset. Custom rules declared with id: foo appear as custom:foo. The directory is read once per process, so a changed file needs a pod restart. Secrets are read from the environment on each read.

The files

One YAML file per concern, named exactly <concern>.yaml (or .yml). Files with other names are skipped with a warning.

FileTop-level keysEntry key
alerts.yamlrules, thresholds, peerdbRules, repeatid (rules, peerdbRules), rule name (thresholds)
routing.yamlroutesid
channels.yamlchannels, webhookTargetschannel, id
quiet-hours.yamlwindowsid
maintenance.yamlwindowsid
digest.yamlenabled, windowMinutesnone (single value)

Every keyed entry carries its key explicitly. The schema is strict: unknown keys are rejected, so a typo is reported instead of silently ignored.

Secrets never go in these files. A ConfigMap is not a Secret. Secret fields take the name of an environment variable (secretEnv, urlEnv, headersEnv), which you back with a Kubernetes Secret. An unset variable skips that entry with a warning that names the variable, never its value. Declared outbound URLs must be HTTPS and pass the same SSRF check as the API. A URL that is itself a credential (Slack or Discord webhook) belongs in a routes entry secretEnv or a webhookTargets urlEnv, not in a channel target.url.

alerts.yaml declares custom rules. metric is a key from the rule builder catalog (for example failed-mutations, readonly-replicas, replication-max-lag, disk-usage-percent). op is one of >, >=, <, <=, and warning and critical must be ordered for that operator.

rules:
  - id: failed-mutations-any
    name: Failed mutations
    metric: failed-mutations
    op: ">="
    warning: 1
    critical: 5

alerts.yaml also accepts a thresholds: map (rule id to warning / critical). The sweep and the current-findings snapshot apply it over the built-in default. It is merged with the HEALTH_THRESHOLD_<RULE>_WARNING and HEALTH_THRESHOLD_<RULE>_CRITICAL environment variables field by field, and the file wins a collision: warning: 70 in the file with HEALTH_THRESHOLD_DISK_USAGE_CRITICAL=95 in env gives 70 / 95. A rule id that is not registered is skipped with a warning in the log. The file is read once per process, so restart after editing it; env changes apply on the next sweep. Only built-in rule ids are honoured by the current-findings snapshot.

thresholds:
  disk-usage:
    warning: 70

alerts.yaml also accepts a repeat: block that paces reminders. It is the file form of HEALTH_ALERT_COOLDOWN_WARNING_MINUTES / HEALTH_ALERT_COOLDOWN_CRITICAL_MINUTES (cooldownMinutes), HEALTH_ALERT_REPEAT_MIN_DELTA and HEALTH_ALERT_REPEAT_MIN_DELTA_<RULE> (minDelta, minDeltaByRule), and HEALTH_ALERT_BACKOFF_MAX_MINUTES (backoffMaxMinutes, must be above 0), and HEALTH_ALERT_LONG_FIRING_DIGEST_HOURS (longFiringDigestHours, must be above 0; off by default). The last one sends one summary per day per channel of alerts that have been open at least that many hours, with the first and the current value. Every key is optional and the file wins over env per key. A per-rule min delta (file, then env) still beats the global one. An invalid block is skipped with a warning and env keeps applying.

repeat:
  cooldownMinutes:
    warning: 120
    critical: 15
  minDelta: 2
  minDeltaByRule:
    disk-usage: 10
  backoffMaxMinutes: 240
  longFiringDigestHours: 24

alerts.yaml also accepts peerdbRules:, per-mirror PeerDB thresholds and mutes. Each entry sets one check (lag, slot-lag, errors, stale-sync, throughput-zero) for the mirrors that match match (matchKind: exact, prefix, or glob, the default). warning must not be above critical. severity: warning stops that check from going past warning. muteUntil (ISO-8601) mutes the matching mirrors until then. Rules from the file show as read-only in Alert Settings. See PeerDB monitoring for how matches are ranked.

peerdbRules:
  - id: fleetreporting-lag
    check: lag
    match: qrep_sg_fleetreporting1_*
    warning: 3600
    critical: 14400
  - id: orders-errors-quiet
    check: errors
    matchKind: exact
    match: pg_orders
    warning: 3
    critical: 20
    severity: warning
    muteUntil: 2026-10-12T06:00:00Z

routing.yaml declares routes. provider is webhook, pagerduty, telegram, ntfy, pushover or matrix; matchRule and matchHost default to * and accept globs. secretEnv names the variable holding the webhook URL, PagerDuty routing key, Telegram bot token, ntfy token (optional), Pushover app token or Matrix access token. target carries the non-secret fields: serviceName (pagerduty), chatId (telegram), url (ntfy), user (pushover), and homeserverUrl plus roomId (matrix).

routes:
  - id: oncall-pagerduty
    provider: pagerduty
    matchRule: "*"
    matchHost: "*"
    minSeverity: critical
    secretEnv: CHM_ROUTE_PAGERDUTY_KEY
    target:
      serviceName: chmonitor-prod
  - id: team-telegram
    provider: telegram
    matchRule: "custom:*"
    secretEnv: CHM_ROUTE_TELEGRAM_TOKEN
    target:
      chatId: "-1001234567890"

channels.yaml declares the server delivery channels (webhook, healthchecks, email, opsgenie, telegram, ntfy, pushover, twilio) and custom webhook targets. A target's urlEnv is required; headersEnv names a variable holding a JSON object of secret headers. repeat: false sends the target the first alert, any escalation and the recovery, but no reminders (default true, reminders on). Precedence against env is per channel, not per field.

channels:
  - channel: telegram
    minSeverity: warning
    secretEnv: HEALTH_CHANNEL_TELEGRAM_TOKEN
    target:
      chatId: "-1001234567890"
  - channel: opsgenie
    secretEnv: HEALTH_CHANNEL_OPSGENIE_KEY
    target:
      region: eu
webhookTargets:
  - id: ops-chat
    name: Ops chat
    format: slack
    minSeverity: warning
    urlEnv: CHM_WEBHOOK_OPS_CHAT_URL
    headersEnv: CHM_WEBHOOK_OPS_CHAT_HEADERS
    repeat: false

quiet-hours.yaml takes days (0 to 6), start and end as HH:mm, an IANA timezone (default UTC) and an optional severityCap: critical.

windows:
  - id: nightly
    days: [1, 2, 3, 4, 5]
    start: "22:00"
    end: "07:00"
    timezone: Asia/Ho_Chi_Minh
    severityCap: critical

maintenance.yaml takes an optional hostId (omit for all hosts), a reason, and ISO-8601 startsAt / endsAt with endsAt later.

windows:
  - id: upgrade-2026-10-05
    hostId: 0
    reason: ClickHouse 25.8 upgrade
    startsAt: "2026-10-05T01:00:00Z"
    endsAt: "2026-10-05T03:00:00Z"

digest.yaml is a single object. windowMinutes is 0 to 1440.

enabled: true
windowMinutes: 15

Fail-closed behaviour

  • A missing directory, or a runtime with no filesystem (Cloudflare Workers), is a silent no-op.
  • Bad YAML, unknown keys, schema violations, entries without a key and duplicate keys are skipped with a warning. The rest of the file and the other files still load, and the pod never fails to start.
  • Warnings name the file and entry, never a field value, so a secret pasted into a ConfigMap by mistake does not reach the logs.
  • An entry whose *Env variable is unset, or whose URL is not an allowed HTTPS endpoint, is skipped.
  • With no database and no declared custom rule, the rule builder still reports "not available" (501) on write.

For the Kubernetes mount, see Alert definitions from a ConfigMap. Variable reference: CHM_HEALTH_CONFIG_DIRECTORY.

Health-sweep cron alerting

The health-sweep endpoint runs checks over all configured hosts and sends a webhook notification when a check meets or exceeds the minimum severity.

Loading diagram…
VariableDefaultDescription
CRON_SECRET(unset = endpoint disabled)Required. Guards GET /api/cron/health-sweep and GET /api/cron/retention-prune. Pass as Authorization: Bearer <secret> only — query-string ?secret= is rejected because it lands in access logs. When it is unset/empty the endpoints fail closed and return HTTP 503 — they do not run. Note that retention-prune is destructive (it deletes conversation rows past each plan's retention window), which is why these routes refuse to run unauthenticated.
CHM_HEALTH_SWEEP_ENABLED(unset = enabled iff CRON_SECRET is set)Gates the scheduled (headless) sweep fired by the Cloudflare Cron trigger. Set true to force-enable, false to pause scheduled sweeps without removing the cron. When unset it fails closed to the CRON_SECRET posture. Does not affect manual curl hits (those only need a valid CRON_SECRET). Truthy values: 1/true/yes/on.
HEALTH_ALERT_ENABLED(optional)Master switch. Unset: alerts dispatch as soon as any channel is configured (webhook URL, email, PagerDuty, and so on); nothing configured stays in dry run. Set false (0/no/off) to pause all dispatch; true/1/yes/on forces it on.
HEALTH_ALERT_WEBHOOK_URL(unset = webhook off)Incoming webhook URL. The channel is auto-detected from the URL: Slack (hooks.slack.com) and Discord (discord.com/api/webhooks) get channel-native rich bodies; Microsoft Teams (*.webhook.office.com, or a Workflows *.logic.azure.com URL) gets an Adaptive Card; Google Chat (chat.googleapis.com) gets a cardsV2 card; any other URL receives the plain {"text": "...", "content": "..."} wrapper. See Alerting to Slack and Discord for setup examples.
HEALTH_ALERT_HEALTHCHECKS_URL(unset = healthchecks disabled)healthchecks.io ping URL. The sweep GETs it on each alert (bare URL) and appends /fail on recovery. Self-hosted or cloud healthchecks.io both work. Overridable from the Server delivery channels UI (see callout below).
HEALTH_ALERT_MIN_SEVERITYwarningGlobal default minimum severity that triggers a notification. Values: warning or critical. Per-channel and per-route overrides below take precedence.
HEALTH_ALERT_<CHANNEL>_MIN_SEVERITY(inherit global)Per-channel severity floor that beats the global default for one channel only (<CHANNEL> ∈ WEBHOOK, EMAIL, OPSGENIE, PAGERDUTY, TELEGRAM, NTFY, PUSHOVER, MATRIX). Example: HEALTH_ALERT_WEBHOOK_MIN_SEVERITY=warning with a critical global sends warnings to the webhook only. Values: warning or critical. Twilio keeps its own HEALTH_ALERT_TWILIO_MIN_SEVERITY.
HEALTH_ALERT_<CHANNEL>_ENABLED(inherit)Set false to silence one channel entirely without unsetting its config (<CHANNEL> as above). HEALTH_ALERT_WEBHOOK_ENABLED=false disables the webhook channel — routes + the legacy global URL — distinct from the master HEALTH_ALERT_ENABLED. Per-channel flags are optional: a configured channel is on by default.
HEALTH_ALERT_TELEGRAM_BOT_TOKEN(unset = Telegram disabled)Telegram Bot API token. Server-only secret — never exposed to the browser. Set together with HEALTH_ALERT_TELEGRAM_CHAT_ID to enable the global Telegram channel; each finding is sent to the chat via the Bot API sendMessage endpoint. Per-rule/per-host Telegram routes (Routing tab) override this fallback.
HEALTH_ALERT_TELEGRAM_CHAT_ID(unset = Telegram disabled)Target Telegram chat id (e.g. -1001234567890, or @channelname). Both this and HEALTH_ALERT_TELEGRAM_BOT_TOKEN must be set for the global Telegram channel to fire.
HEALTH_ALERT_MATRIX_HOMESERVER_URL(unset = native Matrix disabled)Matrix homeserver base URL (e.g. https://matrix.example.org). Set together with HEALTH_ALERT_MATRIX_ACCESS_TOKEN and HEALTH_ALERT_MATRIX_ROOM_ID to post each finding to a room through the Client-Server API. Per-rule/per-host Matrix routes (Routing tab) override this fallback. See Matrix / Element below.
HEALTH_ALERT_MATRIX_ACCESS_TOKEN(unset = native Matrix disabled)Access token of the bot user, sent as Authorization: Bearer <token>. Server-only secret — never exposed to the browser.
HEALTH_ALERT_MATRIX_ROOM_ID(unset = native Matrix disabled)Room id (!abc123:example.org), not an alias.
HEALTH_ALERT_NTFY_URL(unset = ntfy disabled)Full ntfy topic URL (e.g. https://ntfy.sh/my-topic, or a self-hosted server). Each finding is published with Title/Priority/Tags headers + a plain-text body (severity → priority: critical 5/urgent, warning 4/high, recovery 3/default). Per-rule/per-host ntfy routes (Routing tab) override this fallback.
HEALTH_ALERT_NTFY_TOKEN(unset = no auth)Optional ntfy access token for a protected topic, sent as Authorization: Bearer <token>. Server-only secret — never exposed to the browser.
HEALTH_ALERT_TWILIO_ACCOUNT_SID(unset = Twilio disabled)Twilio Account SID. Set together with HEALTH_ALERT_TWILIO_AUTH_TOKEN, HEALTH_ALERT_TWILIO_FROM, and HEALTH_ALERT_TWILIO_TO to enable the global Twilio SMS channel. There are no per-rule/per-host SMS routes, but the credentials are editable from the Server delivery channels UI (see callout below).
HEALTH_ALERT_TWILIO_AUTH_TOKEN(unset = Twilio disabled)Twilio Auth Token. Server-only secret — never exposed to the browser. Authenticates as HTTP Basic (AccountSid:AuthToken) against the Twilio Messages API.
HEALTH_ALERT_TWILIO_FROM(unset = Twilio disabled)The Twilio phone number (E.164, e.g. +15557654321) messages are sent from.
HEALTH_ALERT_TWILIO_TO(unset = Twilio disabled)Comma-separated list of SMS recipients (E.164, e.g. +15551234567,+15559876543). One SMS is sent per recipient per finding.
HEALTH_ALERT_TWILIO_MIN_SEVERITYcriticalMinimum severity that triggers an SMS specifically (independent of HEALTH_ALERT_MIN_SEVERITY above). Values: warning or critical. Defaults to critical because SMS is a last-resort paging channel that costs real money per message — set to warning to also page on warnings.
HEALTH_ALERT_PUSHOVER_TOKEN(unset = Pushover disabled)Pushover application API token. Server-only secret — never exposed to the browser. Set together with HEALTH_ALERT_PUSHOVER_USER to enable the global Pushover channel; each finding is POSTed to the Messages API (severity → priority: critical 2/emergency with retry+expire, warning 0/normal, recovery -1/low-quiet). Per-rule/per-host Pushover routes (Routing tab) override this fallback.
HEALTH_ALERT_PUSHOVER_USER(unset = Pushover disabled)Target Pushover user or group key. Both this and HEALTH_ALERT_PUSHOVER_TOKEN must be set for the global Pushover channel to fire.
HEALTH_ALERT_DIGEST_MINUTES0 (off)Optional time-window digest for alert batching. 0/unset dispatches every finding on the sweep tick (in-pass grouping still applies — see the callout below). A positive value buffers non-critical findings and flushes them together once the window elapses, collapsing a burst into one message per target; critical findings always bypass the buffer and page immediately. Requires a D1 binding for the buffer; with none it degrades to 0 (off). Overridable from Health → Settings → Alerts → Digest batching.
HEALTH_ALERT_INSTANCE_NAME(unset)Deployment label, e.g. prod-eu. Every alert and digest starts with [prod-eu]; Hookshot also uses it as the sender name. Use it when several deployments post into the same channel.
HEALTH_ALERT_DASHBOARD_URL(unset)Public http(s) base URL of this dashboard. Alerts add an Open dashboard link to <url>/health?host=<id> (Slack button, Discord embed link, Matrix/Hookshot/Google Chat/Teams/Telegram link, dashboardUrl in raw JSON). The cron sweep has no request origin, so this must be set.
HEALTH_ALERT_METADATA(unset)Non-secret key/values, key=value,key2=value2 or a JSON object (max 10 pairs, values ≤128 chars). Shown with the instance name in a context line, sent as metadata in raw JSON, and available to custom templates as {{meta.<key>}}.

Built-in alert rules

The sweep evaluates these rules on every host. Thresholds are the defaults; override them per rule from the UI, alerts.yaml thresholds:, or HEALTH_THRESHOLD_<RULE>_WARNING / _CRITICAL. A rule whose system table is missing (for example no system.backup_log) is skipped, not failed.

backups

  • Failed Backups (24h)failed-backupswarning ≥ 1critical ≥ 3

    Backup operations that ended in FAILED status in the last 24 hours.

    Requires
    system.backup_log (skipped when missing)
    Alert reads
    1 failed backup
    • Remote storage unavailable or credentials expired
    • Disk full on backup destination
    • Network interruption during backup transfer
    Rule SQL
    SELECT count() AS failed_count
    FROM system.backup_log
    WHERE event_time_microseconds > now() - INTERVAL 24 HOUR
      AND status IN ('BACKUP_FAILED', 'RESTORE_FAILED')

dictionaries

  • Dictionary Load Failuresdictionary-failureswarning ≥ 1critical ≥ 5

    Dictionaries whose last load or reload failed. A dictionary that never loaded breaks every dictGet on it; one whose reload failed keeps serving stale data. Fix the source named in last_exception, then SYSTEM RELOAD DICTIONARY.

    Requires
    system.dictionaries (skipped when missing)
    Alert reads
    1 failing dictionary
    Rule SQL
    SELECT countIf(last_exception != '') AS failed_count
    FROM system.dictionaries
    Get failing dictionaries
    SELECT database, name, status, last_successful_update_time, substring(last_exception, 1, 300) AS last_exception
    FROM system.dictionaries
    WHERE last_exception != ''
    ORDER BY database, name
    LIMIT 20

logs

  • Keeper Exceptionskeeper-unavailablewarning ≥ 5critical ≥ 20

    Recent KEEPER_EXCEPTION events. Sustained exceptions indicate quorum issues.

    Requires
    system.error_log (skipped when missing)
    Alert reads
    5 exceptions
    Rule SQL
    SELECT coalesce(max(value) - min(value), 0) AS exception_count
    FROM merge('system', '^error_log')
    WHERE error = 'KEEPER_EXCEPTION'
      AND event_time > now() - INTERVAL 1 HOUR
  • Fatal Log Entriesfatal-log-entrieswarning ≥ 1critical ≥ 5

    Fatal errors in the server text log in the last hour.

    Requires
    system.text_log (skipped when missing)
    Alert reads
    1 fatal log entries
    Rule SQL
    SELECT count() AS fatal_count
    FROM system.text_log
    WHERE level = 'Fatal'
      AND event_time >= now() - INTERVAL 1 HOUR

other

  • Concurrent Query Saturationconcurrent-querieswarning ≥ 80critical ≥ 95

    Running queries as a percentage of max_concurrent_queries. At 100% every new query fails with TOO_MANY_SIMULTANEOUS_QUERIES — find what is holding slots (long or stuck queries, a client retry storm).

    Requires
    system.server_settings (skipped when missing)
    Alert reads
    80% of max_concurrent_queries
    Rule SQL
    SELECT round(
      (SELECT value FROM system.metrics WHERE metric = 'Query') * 100.0
      / nullIf(toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_concurrent_queries')), 0),
      1
    ) AS concurrency_percent
    Get running queries by user
    SELECT user, count() AS running, round(max(elapsed)) AS oldest_seconds, any(substring(query, 1, 200)) AS sample_query
    FROM system.processes
    GROUP BY user
    ORDER BY running DESC
    LIMIT 20
  • Delayed Insertsdelayed-insertswarning ≥ 1critical ≥ 5

    INSERTs currently throttled because a partition exceeded parts_to_delay_insert. Merges are falling behind inserts — batch inserts into fewer, larger blocks or coarsen PARTITION BY.

    Requires
    system.metrics (skipped when missing)
    Alert reads
    1 delayed insert
    • Too-small batches producing many parts (recommend ≥ 1000 rows / insert)
    • Background merges starved (low background pool size, slow disk)
    • `parts_to_delay_insert` / `parts_to_throw_insert` tuned too low
    Rule SQL
    SELECT value AS delayed_inserts
    FROM system.metrics
    WHERE metric = 'DelayedInserts'
    Get partitions with the most parts
    SELECT database, table, partition, count() AS parts
    FROM system.parts
    WHERE active
    GROUP BY database, table, partition
    ORDER BY parts DESC
    LIMIT 20
  • Memory Pressurememory-pressurewarning ≥ 80critical ≥ 90

    Server memory tracked by ClickHouse (MemoryTracking) as a percentage of the server memory limit (max_server_memory_usage). Near 100% queries fail with MEMORY_LIMIT_EXCEEDED — find the heaviest queries and cap max_memory_usage.

    Requires
    system.asynchronous_metrics (skipped when missing)
    Alert reads
    80% of the server memory limit
    Rule SQL
    SELECT round(
      (SELECT value FROM system.metrics WHERE metric = 'MemoryTracking') * 100.0
      / nullIf(coalesce(
        nullIf(toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage')), 0),
        (SELECT value FROM system.asynchronous_metrics WHERE metric = 'OSMemoryTotal')
          * toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage_to_ram_ratio'))
      ), 0),
      1
    ) AS memory_percent
    Get top queries by memory
    SELECT query_id, user, elapsed, formatReadableSize(memory_usage) AS memory, substring(query, 1, 200) AS query
    FROM system.processes
    ORDER BY memory_usage DESC
    LIMIT 20
  • Long-Running Querylong-running-querieswarning ≥ 600critical ≥ 3600

    Seconds the longest running query has been running (system.processes, initial queries only), naming up to 5 as user:query_id. A monitoring signal: shown on /health and fed to "Runaway query under system pressure", never sent to a channel on its own.

    Alert reads
    longest query running 600s
    • Missing PRIMARY KEY / sort order for the query pattern
    • Cross-shard JOINs without distributed_product_mode tuning
    • Stuck query waiting on lock or external resource
    Rule SQL
    SELECT round(max(elapsed)) AS longest_seconds,
      arrayMap(t -> concat(t.2, ':', t.3),
        arraySlice(arrayReverseSort(groupArray((elapsed, user, query_id))), 1, 5)) AS longest_queries
    FROM system.processes
    WHERE is_initial_query = 1 AND query_id != queryID()
    Get longest running queries
    SELECT query_id, user, round(elapsed) AS elapsed_seconds, formatReadableSize(memory_usage) AS memory, substring(query, 1, 200) AS query
    FROM system.processes
    WHERE is_initial_query = 1
    ORDER BY elapsed DESC
    LIMIT 20
  • Heavy Query Memoryheavy-query-memorywarning ≥ 30critical ≥ 50

    Memory of the single heaviest running query as a percentage of the server memory limit (the same limit Memory Pressure uses), naming up to 5 as user:query_id. A monitoring signal: shown on /health and fed to "Runaway query under system pressure", never sent to a channel on its own.

    Requires
    system.asynchronous_metrics (skipped when missing)
    Alert reads
    heaviest query uses 30% of the server memory limit
    Rule SQL
    SELECT round(
      max(memory_usage) * 100.0
      / nullIf(coalesce(
        nullIf(toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage')), 0),
        (SELECT value FROM system.asynchronous_metrics WHERE metric = 'OSMemoryTotal')
          * toFloat64OrZero((SELECT value FROM system.server_settings WHERE name = 'max_server_memory_usage_to_ram_ratio'))
      ), 0),
      1
    ) AS heaviest_query_memory_percent,
      arrayMap(t -> concat(t.2, ':', t.3),
        arraySlice(arrayReverseSort(groupArray((memory_usage, user, query_id))), 1, 5)) AS heaviest_queries
    FROM system.processes
    WHERE is_initial_query = 1 AND query_id != queryID()
    Get top queries by memory
    SELECT query_id, user, round(elapsed) AS elapsed_seconds, formatReadableSize(memory_usage) AS memory, substring(query, 1, 200) AS query
    FROM system.processes
    WHERE is_initial_query = 1
    ORDER BY memory_usage DESC
    LIMIT 20
  • Broken Detached Partsbroken-detached-partswarning ≥ 1critical ≥ 1

    Parts ClickHouse detached because they were broken (checksum mismatch, corruption, unexpected files). On a replica the data is usually refetched; on a non-replicated table it may be lost — inspect before dropping.

    Requires
    system.detached_parts (skipped when missing)
    Alert reads
    1 broken detached part
    Rule SQL
    SELECT count() AS broken_count,
      groupArray(5)(concat(database, '.', table, ' ', name)) AS affected
    FROM system.detached_parts
    WHERE reason LIKE 'broken%'
    Get broken detached parts
    SELECT database, table, name, reason
    FROM system.detached_parts
    WHERE reason LIKE 'broken%'
    ORDER BY database, table, name
    LIMIT 50
  • Detached Parts To Reviewdetached-parts-reviewwarning ≥ 1critical ≥ 1000000

    Off by default; enable with HEALTH_ALERT_ENABLE_RULES=detached-parts-review. Detached parts hold disk space until someone reviews them. The value is the larger of two ratios: detached bytes against 1% of total disk space, and the oldest detached part against 30 days. 1 means one limit is reached. Coverage is per replica: this host only, a part detached here says nothing about the other replicas, so check each replica. Open the Detached Parts Triage page (/detached-parts-triage) to review each part before acting.

    Requires
    system.detached_parts (skipped when missing)
    Alert reads
    detached parts at 1.00x the review limit (1% of disk or 30 days)
    Rule SQL
    SELECT if(count() = 0, 0, greatest(
      ifNull(sum(bytes_on_disk) * 100.0 / nullIf((SELECT sum(total_space) FROM system.disks), 0), 0),
      ifNull((toUnixTimestamp(now()) - toUnixTimestamp(min(modification_time))) / 86400.0 / 30, 0)
    )) AS review_ratio
    FROM system.detached_parts
    List detached parts to review
    SELECT database, table, name, reason, bytes_on_disk, modification_time
    FROM system.detached_parts
    ORDER BY modification_time ASC, bytes_on_disk DESC
    LIMIT 50

parts & merges

  • Failed Mutationsfailed-mutationswarning ≥ 1critical ≥ 5

    Mutations that are not complete and have recorded a failure. Failed mutations block subsequent mutations on the same table.

    Requires
    system.mutations (skipped when missing)
    Alert reads
    1 failed mutation
    • Mutation references a column that no longer exists
    • Disk full on a replica preventing part writes
    • Hardware / network fault during part rewrite
    Rule SQL
    SELECT countIf(is_done = 0 AND latest_fail_reason != '') AS failed_count,
      groupArrayIf(5)(concat(database, '.', table, ' ', mutation_id), is_done = 0 AND latest_fail_reason != '') AS affected
    FROM system.mutations
    Get failed mutations
    SELECT database, table, mutation_id, command, latest_fail_reason, latest_fail_time
    FROM system.mutations
    WHERE is_done = 0 AND latest_fail_reason != ''
    ORDER BY latest_fail_time DESC
    LIMIT 20
  • Stuck Mergesstuck-mergeswarning ≥ 1critical ≥ 3

    Merges running for more than 10 minutes. Stuck merges block table inserts and consume resources.

    Requires
    system.merges (skipped when missing)
    Alert reads
    1 stuck merge
    • Very large parts requiring many hours to merge
    • Disk I/O saturation or slow storage
    • Insufficient background_pool_size
    Rule SQL
    SELECT count() AS stuck_count,
      groupArray(5)(concat(database, '.', table, ' ', result_part_name)) AS affected
    FROM system.merges
    WHERE elapsed > 600
    Get stuck merges
    SELECT database, table, elapsed, progress, num_parts, total_size_bytes_compressed
    FROM system.merges
    WHERE elapsed > 600
    ORDER BY elapsed DESC
    LIMIT 20
  • Parts Pressureparts-pressurewarning ≥ 80critical ≥ 95

    Worst partition’s active parts as a percentage of parts_to_throw_insert. A predictive "too many parts" signal — sustained pressure leads to throttled then rejected inserts.

    Requires
    system.parts (skipped when missing)
    Alert reads
    80% of parts_to_throw_insert
    • Frequent small inserts producing parts faster than merges consume them
    • Background merges starved (low background_pool_size, slow disk)
    • Partition key too granular, spreading inserts across many partitions
    • `parts_to_throw_insert` / `parts_to_delay_insert` lowered per table
    Rule SQL
    WITH parts_now AS (
        SELECT database, table, partition_id, partition, count() AS parts
        FROM system.parts
        WHERE active AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
        GROUP BY database, table, partition_id, partition
      ),
      table_overrides AS (
        SELECT
          database,
          name AS table,
          toInt64OrZero(extract(replaceRegexpOne(
          replaceRegexpAll(create_table_query, '\\x27(?:[^\\x27\\\\]|\\\\.)*\\x27', ''),
          '(?s)^.*\\sSETTINGS\\s+',
          ''
        ), '(?:^|,)\\s*parts_to_throw_insert\\s*=\\s*([0-9]+)')) AS throw_override,
          toInt64OrZero(extract(replaceRegexpOne(
          replaceRegexpAll(create_table_query, '\\x27(?:[^\\x27\\\\]|\\\\.)*\\x27', ''),
          '(?s)^.*\\sSETTINGS\\s+',
          ''
        ), '(?:^|,)\\s*parts_to_delay_insert\\s*=\\s*([0-9]+)')) AS delay_override
        FROM system.tables
        WHERE database IN (SELECT name FROM system.databases WHERE engine IN ('Atomic', 'Lazy', 'Memory', 'Ordinary', 'Replicated', 'Shared'))
          AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
          AND (database, name) IN (SELECT database, table FROM parts_now)
      ),
      server_defaults AS (
        SELECT
          coalesce((SELECT toInt64OrNull(value) FROM system.merge_tree_settings WHERE name = 'parts_to_throw_insert'), 3000) AS throw_default,
          coalesce((SELECT toInt64OrNull(value) FROM system.merge_tree_settings WHERE name = 'parts_to_delay_insert'), 1000) AS delay_default
      ),
      limits AS (
        SELECT
          p.database AS database,
          p.table AS table,
          p.partition_id AS partition_id,
          p.partition AS partition,
          p.parts AS parts,
          if(o.throw_override > 0, o.throw_override, (SELECT throw_default FROM server_defaults)) AS throw_limit,
          if(o.delay_override > 0, o.delay_override, (SELECT delay_default FROM server_defaults)) AS delay_limit
        FROM parts_now p
        LEFT JOIN table_overrides o ON p.database = o.database AND p.table = o.table
        WHERE p.parts >= 20
      )
    SELECT round(max(parts * 100.0 / nullIf(throw_limit, 0)), 1) AS pressure_percent
    FROM limits
    Get partitions near the throw limit
    SELECT database, table, partition, count() AS parts
    FROM system.parts
    WHERE active
    GROUP BY database, table, partition
    ORDER BY parts DESC
    LIMIT 20
  • TTL & Partition Healthttl-partition-healthwarning ≥ 1critical ≥ 5

    MergeTree tables with partition bloat, a time-based PARTITION BY and no table TTL, or a merge backlog. Recommend-only — never applies ALTER TTL or DROP PARTITION.

    Requires
    system.parts (skipped when missing)
    Alert reads
    1 table to review
    • PARTITION BY too granular (hourly or high-cardinality keys)
    • Time-series table with no table-level TTL, so old partitions never drop
    • Inserts producing parts faster than background merges consume them
    Rule SQL
    WITH mergetree_tables AS (
          SELECT
            database,
            name,
            engine,
            partition_key,
            engine_full
          FROM system.tables
          WHERE is_temporary = 0
            AND database IN (SELECT name FROM system.databases WHERE engine IN ('Atomic', 'Lazy', 'Memory', 'Ordinary', 'Replicated', 'Shared'))
            AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
            AND positionCaseInsensitive(engine, 'MergeTree') > 0
        ),
        part_stats AS (
          SELECT
            database,
            table,
            uniqExact(partition) AS partitions,
            count() AS active_parts,
            if(
              uniqExact(partition) = 0,
              0,
              round(count() / uniqExact(partition), 2)
            ) AS parts_per_partition,
            sum(bytes_on_disk) AS bytes_on_disk,
            sum(rows) AS rows
          FROM system.parts
          WHERE active
            AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
          GROUP BY database, table
        ),
        inventory AS (
          SELECT
            t.database,
            t.name AS table,
            concat(t.database, '.', t.name) AS full_table,
            t.database AS _database,
            t.name AS _table,
            t.engine,
            t.partition_key,
            
          if(
            positionCaseInsensitive(t.engine_full, ' TTL ') > 0,
            trim(BOTH ' ' FROM replaceRegexpOne(
              substring(
                t.engine_full,
                positionCaseInsensitive(t.engine_full, ' TTL ') + 5
              ),
              '\\s+SETTINGS\\s+.*$',
              ''
            )),
            ''
          ) AS ttl_expression,
            
          multiIf(
            ifNull(p.partitions, 0) >= 1000,
              'Rebuild with coarser PARTITION BY',
            ifNull(p.partitions, 0) >= 500,
              'Consider monthly partitions',
            (
            positionCaseInsensitive(t.partition_key, 'toYYYYMM') > 0
            OR positionCaseInsensitive(t.partition_key, 'toStartOf') > 0
            OR positionCaseInsensitive(t.partition_key, 'toDate') > 0
            OR positionCaseInsensitive(t.partition_key, 'toMonday') > 0
            OR positionCaseInsensitive(t.partition_key, 'toYearWeek') > 0
            OR positionCaseInsensitive(t.partition_key, 'toISOWeek') > 0
          )
              AND positionCaseInsensitive(t.engine_full, ' TTL ') = 0,
              'Add table TTL',
            ifNull(p.partitions, 0) > 0
              AND ifNull(p.active_parts, 0) / ifNull(p.partitions, 1)
                >= 10,
              'Check merge backlog',
            ''
          ) AS recommendation,
            ifNull(p.partitions, 0) AS partitions,
            ifNull(p.active_parts, 0) AS active_parts,
            ifNull(p.parts_per_partition, 0) AS parts_per_partition,
            ifNull(p.bytes_on_disk, 0) AS bytes_on_disk,
            ifNull(p.rows, 0) AS rows
          FROM mergetree_tables AS t
          LEFT JOIN part_stats AS p
            ON p.database = t.database AND p.table = t.name
        )
    SELECT count() AS flagged_count
    FROM inventory
    WHERE recommendation != ''
    Get tables with 500+ partitions
    SELECT
      concat(database, '.', table) AS full_table,
      uniqExact(partition) AS partitions,
      count() AS active_parts
    FROM system.parts
    WHERE active
    GROUP BY database, table
    HAVING partitions >= 500
    ORDER BY partitions DESC
    LIMIT 20

queries

  • Query Timeout Breaches (1h)query-timeoutwarning ≥ 1critical ≥ 10

    Queries killed due to timeout (TIMEOUT_EXCEEDED) in the last hour.

    Requires
    system.query_log (skipped when missing)
    Alert reads
    1 timeout kills in last hour
    • max_execution_time set too low for heavy analytical queries
    • Missing sort key for the filter predicate
    • Cross-shard queries without proper distribution settings
    Rule SQL
    SELECT count() AS timeout_count,
      groupArray(5)(query_id) AS affected
    FROM system.query_log
    WHERE event_time > now() - INTERVAL 1 HOUR
      AND type IN ('ExceptionWhileProcessing', 'ExceptionBeforeStart')
      AND (exception_code = 159 OR exception LIKE '%TIMEOUT_EXCEEDED%')
  • Rejected Inserts (1h)rejected-insertswarning ≥ 1critical ≥ 1

    INSERTs rejected with TOO_MANY_PARTS in the last hour. Data was not written — the client must retry. Reduce insert frequency or coarsen PARTITION BY.

    Requires
    system.query_log (skipped when missing)
    Alert reads
    1 rejected inserts in last hour
    Rule SQL
    SELECT count() AS rejected_count,
      groupArray(5)(query_id) AS affected
    FROM system.query_log
    WHERE event_time > now() - INTERVAL 1 HOUR
      AND type IN ('ExceptionBeforeStart', 'ExceptionWhileProcessing')
      AND exception_code = 252

replication

  • Readonly Replicasreadonly-replicaswarning ≥ 1critical ≥ 3

    Replicas in read-only mode cannot accept writes.

    Requires
    system.replicas (skipped when missing)
    Alert reads
    1 readonly replica
    • ZooKeeper / ClickHouse Keeper unavailable or laggy
    • Replica metadata corruption (force_restore_data required)
    • Disk full on the replica
    Rule SQL
    SELECT count() AS readonly_count,
      groupArray(5)(concat(database, '.', table)) AS affected
    FROM system.replicas
    WHERE is_readonly = 1
  • Replication Lagreplication-lagwarning ≥ 30critical ≥ 300

    Maximum absolute_delay across all replicas (seconds behind leader).

    Requires
    system.replicas (skipped when missing)
    Alert reads
    30s max delay
    • Slow network or saturated inter-server connections
    • Large fetch backlog in `replication_queue`
    • ZooKeeper / Keeper latency
    Rule SQL
    SELECT max(absolute_delay) AS max_lag
    FROM system.replicas
    Get lagging replicas
    SELECT database, table, replica_name, absolute_delay, is_readonly
    FROM system.replicas
    ORDER BY absolute_delay DESC
    LIMIT 20
  • Stuck Replication Queuereplication-queue-stuckwarning ≥ 1critical ≥ 10

    Replication queue entries retried more than 100 times or queued for over an hour. A stuck entry stops the replica from catching up and grows replication lag.

    Requires
    system.replication_queue (skipped when missing)
    Alert reads
    1 stuck queue entry
    Rule SQL
    SELECT count() AS stuck_count,
      groupArray(5)(concat(database, '.', table, ' ', node_name)) AS affected
    FROM system.replication_queue
    WHERE num_tries > 100 OR create_time < now() - INTERVAL 1 HOUR
    Get stuck queue entries
    SELECT database, table, type, create_time, num_tries, last_exception, postpone_reason
    FROM system.replication_queue
    WHERE num_tries > 100 OR create_time < now() - INTERVAL 1 HOUR
    ORDER BY create_time ASC
    LIMIT 20
  • Replica Keeper Session Expiredreplica-session-expiredwarning ≥ 1critical ≥ 3

    Replicated tables whose ZooKeeper/Keeper session has expired. These tables are read-only until the session is re-established — check Keeper health and network to the Keeper ensemble.

    Requires
    system.replicas (skipped when missing)
    Alert reads
    1 expired replica session
    Rule SQL
    SELECT countIf(is_session_expired) AS expired_count,
      groupArrayIf(5)(concat(database, '.', table), is_session_expired) AS affected
    FROM system.replicas
    Get replicas with an expired session
    SELECT database, table, replica_name, is_readonly, zookeeper_path
    FROM system.replicas
    WHERE is_session_expired
    LIMIT 20
  • Stuck Distributed Send Queuedistribution-queuewarning ≥ 1critical ≥ 5

    Distributed tables whose async insert queue is not draining: queued files with send errors or a blocked queue, or broken files that were never sent. The data is not on the shards yet, and the queue grows on local disk — check the shard named in last_exception.

    Requires
    system.distribution_queue (skipped when missing)
    Alert reads
    1 Distributed table with a stuck queue
    Rule SQL
    SELECT countIf((data_files > 0 AND (error_count > 0 OR is_blocked)) OR broken_data_files > 0) AS stuck_tables
    FROM system.distribution_queue
    Get stuck Distributed queues
    SELECT database, table, is_blocked, error_count, data_files, formatReadableSize(data_compressed_bytes) AS queued, broken_data_files, last_exception_time, substring(last_exception, 1, 300) AS last_exception
    FROM system.distribution_queue
    WHERE data_files > 0 OR broken_data_files > 0
    ORDER BY error_count DESC, data_files DESC
    LIMIT 20

storage

  • Disk Usagedisk-usagewarning ≥ 80critical ≥ 95

    Worst-case disk utilization across all configured volumes.

    Requires
    system.disks (skipped when missing)
    Alert reads
    80% used (worst disk)
    Rule SQL
    SELECT round(max((total_space - free_space) * 100.0 / nullIf(total_space, 0)), 1) AS disk_percent
    FROM system.disks
  • Disk Fill Forecastdisk-time-to-fullwarning ≥ 100critical ≥ 400

    Fits a linear trend to each disk’s free space over the last 6 hours and reports how much of the current free space that trend uses up in 24 hours. 100% means the disk fills within a day, 400% within 6 hours. Catches a fast-filling disk long before disk-usage crosses its percent threshold.

    Requires
    system.asynchronous_metric_log (skipped when missing)
    Alert reads
    full in ~24h at the current rate
    Rule SQL
    SELECT round(maxOrNull(fill_pct), 1) AS fill_24h_percent
    FROM (
      SELECT
        argMax(value, event_time) AS free_now,
        simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 AS bytes_per_second,
        if(free_now <= 0, 10000, least(greatest(-bytes_per_second, 0) * 86400 * 100 / free_now, 10000)) AS fill_pct
      FROM system.asynchronous_metric_log
      WHERE startsWith(metric, 'DiskAvailable_')
        AND event_date >= toDate(now() - INTERVAL 6 HOUR)
        AND event_time > now() - INTERVAL 6 HOUR
      GROUP BY metric
      HAVING max(event_time) - min(event_time) >= 3 * 3600 AND count() >= 30
    )
    Get disk fill rates
    SELECT
      replaceOne(metric, 'DiskAvailable_', '') AS disk,
      formatReadableSize(argMax(value, event_time)) AS free_now,
      formatReadableSize(-simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 * 3600) AS used_per_hour,
      if(simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 < 0,
        round(argMax(value, event_time) / -simpleLinearRegression(toFloat64(toUInt32(event_time)), value).1 / 3600, 1),
        NULL) AS hours_to_full
    FROM system.asynchronous_metric_log
    WHERE startsWith(metric, 'DiskAvailable_')
      AND event_date >= toDate(now() - INTERVAL 6 HOUR)
      AND event_time > now() - INTERVAL 6 HOUR
    GROUP BY metric
    ORDER BY hours_to_full ASC NULLS LAST
    LIMIT 20

views

  • MV Refresh Failuresmv-refresh-failureswarning ≥ 1critical ≥ 3

    Materialized views with REFRESH schedule that have failed or errored their last refresh cycle.

    Requires
    system.view_refreshes (skipped when missing)
    Alert reads
    1 failed MV refresh
    • Source table schema changed incompatibly
    • MV query references a missing table or column
    • Resource exhaustion during refresh
    Rule SQL
    SELECT countIf(exception != '' OR retry > 0) AS failed_count,
      groupArrayIf(5)(concat(database, '.', view), exception != '' OR retry > 0) AS affected
    FROM system.view_refreshes

compound

  • Replica Split-Brain Riskreplica-split-brain

    Replication lag AND readonly replicas both firing at once — a stronger signal of a stuck/diverging replica than either metric alone.

    Fires when all of: replication-lag, readonly-replicas

  • Merge Pressuremerge-pressure

    Stuck merges AND high disk usage both firing at once — merges are likely stalled fighting for disk headroom rather than transient load.

    Fires when all of: stuck-merges, disk-usage

  • Runaway Query Under System Pressurerunaway-query-impact

    A long-running or memory-heavy query (at least warning) while the server is critical on memory or concurrent query slots. The only page for heavy queries: a heavy query on a healthy server stays a signal on /health.

    Fires when all of: long-running-queries, heavy-query-memory, memory-pressure, concurrent-queries

Rules that report a count also name what they count, so the alert says which tables, parts, mutations, views or queries to look at. The label lists up to 5 names, then "and N more", for example 7 stuck merges: db.events all_1_9_2, db.logs all_3_7_1 and 5 more. The names come from the same query (groupArray(5)), so they add no extra statement. Each name is cut to 80 characters and escaped in HTML, Slack and Teams bodies.

keeper-unavailable warns at 5, not 1: one KEEPER_EXCEPTION an hour is routine (a session reconnect or leader re-election). Sustained exceptions are the quorum signal. rejected-inserts reads a one-hour query_log window because the RejectedInserts event in system.events is cumulative since server start and would keep firing after the incident ended. A rejected insert or a broken part is possible data loss, so both go straight to critical. memory-pressure divides tracked memory by max_server_memory_usage, the limit at which queries fail, not by host RAM. In a container that limit follows the cgroup memory limit.

distribution-queue counts Distributed tables whose async insert queue has send errors, is blocked, or holds broken files. A queue that drains normally never counts, so a shard outage is the usual cause: the data waits on the sender's disk until the shard is back.

concurrent-queries compares running queries with max_concurrent_queries and is skipped when the limit is 0 (unlimited). At 100% the sweep's own query is refused too, so a saturated host shows a failed check rather than this alert; the 80% and 95% thresholds are the warning before that.

dictionary-failures counts dictionaries with a last_exception. That includes a reload that failed on a dictionary still marked LOADED, which keeps answering dictGet with stale data. A dictionary that was never loaded (lazy loading) has no exception and does not count.

disk-time-to-full fits a straight line to each disk's free space over the last 6 hours of system.asynchronous_metric_log and reports how much of the free space that trend uses in 24 hours: 100 means full within a day (warning), 400 within 6 hours (critical). A disk that is not shrinking reads 0. It needs asynchronous_metric_log enabled and at least 3 hours of samples per disk; until then it reports nothing. disk-usage still alerts on percent used.

long-running-queries reports how long the longest running query has been running (warning 10 minutes, critical 1 hour). heavy-query-memory reports the memory of the heaviest running query as a percentage of the same server memory limit memory-pressure uses (warning 30%, critical 50%). Both name up to 5 queries as user:query_id, longest or heaviest first. Both are signals, not pages: they show on the Health page but never send an alert on their own. A long report query on a healthy server is normal and should not wake anyone.

runaway-query-impact is the page for heavy queries. It fires, always as critical, only when a long or heavy query (at least warning) runs while the server itself is critical: memory-pressure or concurrent-queries is critical. Put simply, a heavy query alone is shown, and it pages only when the whole system is in trouble at the same time. The alert names the offending queries so you can kill the right one. To also get alerts for one of the two signals on its own, list it in HEALTH_ALERT_NOTIFY_RULES.

Alert grouping & digest batching

When one sweep finds several alerts bound for the same delivery target (e.g. disk filling on 8 hosts, all routed to one Slack channel), they are combined into one message — a summary line (3 critical, 2 warning on 4 hosts) followed by one row per rule and severity — instead of N separate pings. Findings of the same rule share a row with the rule title, a count and up to three example names ([WARNING] Source stopped sending rows · 34 (source-1, source-2, source-3, +31)); a rule with one finding keeps its full line. When several findings share one host the header names it (on 1 host (node-1)) and the rows leave it out. A sweep that only adds findings of a rule that is already open, or only resolves, sends one short line to Slack and generic webhook targets (+1 warning · 1 resolved · 60 open) instead of the whole table; the first finding of a rule, an escalation and a reminder still send their normal message. Matrix and Hookshot already list findings under a heading per rule, and the generic webhook still carries every finding in alerts. A reminder for a condition that is still firing takes one short line (still open, 3 h (82.8 -> 82.9), with how the value moved since the last notification when both values are numbers) instead of the full detail, also when it is the only finding. A Slack message with ack buttons keeps its card. This in-pass grouping is always on and needs no configuration; it applies to the generic webhook, Slack, and Telegram channels, to native Matrix rooms, and to Matrix Hookshot and Element / Matrix custom webhook targets that have no title or body template (every other channel keeps per-finding sends). Native Matrix rooms and Hookshot / Matrix targets are always grouped within the sweep: the time-window digest never buffers them, because they carry secrets (an access token or headers). To also batch across time, enable the optional time-window digest (HEALTH_ALERT_DIGEST_MINUTES or the Alerts tab): non-critical findings wait up to the window and flush together, while criticals still page immediately.

Edit channels from the UI (no redeploy)

Every channel above is also editable from Health → Settings → Alerts → Server delivery channels. A channel saved there is persisted per-owner (in D1, when configured) and takes precedence over its HEALTH_ALERT_* environment variable on the next sweep — so you can add or rotate a destination without a redeploy. Leaving a secret field blank on save keeps the stored secret. A channel with no saved config falls back to its env variable (shown as "Configured via server env"), and a deployment with no D1 binding simply keeps using the env variables. Browser notifications stay per-browser (local to your session).

SMS costs real money

Every Twilio dispatch is a billed SMS — one message per configured recipient, per finding. HEALTH_ALERT_TWILIO_MIN_SEVERITY defaults to critical so routine warnings never page a phone; only raise it to warning if you understand the cost. Message bodies are also truncated to Twilio's 1600-character SMS limit.

CRON_SECRET is required

CRON_SECRET is required for the cron endpoints. When it is unset, /api/cron/health-sweep and /api/cron/retention-prune fail closed and return HTTP 503 (they do not run) — retention-prune is destructive, so it must never be reachable unauthenticated. Set CRON_SECRET and pass it as Authorization: Bearer <secret> from your cron caller.

Set secrets

Example for Cloudflare Workers (using wrangler secret put):

wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URL
# Optional: Telegram channel (bot token is a secret; chat id is a plain var)
wrangler secret put HEALTH_ALERT_TELEGRAM_BOT_TOKEN
# Optional: ntfy channel (topic URL is a plain var; access token is a secret)
wrangler secret put HEALTH_ALERT_NTFY_TOKEN
# Optional: Twilio SMS channel (auth token is a secret; SID/from/to are plain vars)
wrangler secret put HEALTH_ALERT_TWILIO_AUTH_TOKEN
# Optional: Pushover channel (app token is a secret; user key is a plain var)
wrangler secret put HEALTH_ALERT_PUSHOVER_TOKEN
# Optional: native Matrix (access token is a secret; homeserver URL and room id are plain vars)
wrangler secret put HEALTH_ALERT_MATRIX_ACCESS_TOKEN

Enable alerting

Example environment block:

HEALTH_ALERT_MIN_SEVERITY=warning
HEALTH_ALERT_WEBHOOK_URL=https://hooks.slack.com/services/...
CRON_SECRET=<random-secret>

Call it manually

curl -H "Authorization: Bearer $CRON_SECRET" \
  https://your-chmonitor.example.com/api/cron/health-sweep

The endpoint returns a JSON array of check results. It always returns HTTP 200; alert dispatch happens server-side.

Matrix / Element

chmonitor can post alerts to a Matrix room itself, through the Matrix Client-Server API. It sends each alert as an m.notice event with PUT /_matrix/client/v3/rooms/{roomId}/send/m.room.message/{txnId}. No bridge is needed.

  1. Create a bot user on your homeserver, for example @chmonitor:example.org.
  2. Get its access token. Sign in as the bot in Element and copy the token from Settings → Help & About → Access token, or call POST /_matrix/client/v3/login.
  3. Invite the bot to the room, then sign in as the bot and accept the invite. The bot must already be joined: chmonitor does not join rooms.
  4. Copy the room id (!abc123:example.org) from Room settings → Advanced in Element. A room alias (#ops:example.org) does not work.
  5. Set HEALTH_ALERT_MATRIX_HOMESERVER_URL, HEALTH_ALERT_MATRIX_ACCESS_TOKEN and HEALTH_ALERT_MATRIX_ROOM_ID, or add a Matrix / Element room route on the Routing tab and click Send test message.

A homeserver URL saved from the Routing tab, or set in a routing.yaml target, must be public HTTPS with no username, password, query or fragment; private and internal addresses are refused, both when you save it and again at every send. It is stored without trailing slashes (a path prefix such as /matrix is kept). The env URL is set by the operator and may be http and point at a LAN homeserver; an env URL or room id that is not valid disables native Matrix and logs one warning. Redirects are never followed, so the access token only goes to the host you configured.

Encrypted rooms

chmonitor sends events unencrypted. In an end-to-end encrypted room the alerts still arrive, but they are not encrypted. Use a room without encryption for alerts, or accept that alert text is readable by the homeserver.

Native Matrix or Hookshot? A Hookshot (or other bridge) webhook is a custom webhook target with the hookshot or matrix format: chmonitor POSTs to the bridge URL and the bridge posts into the room, and findings from one sweep are grouped into one message. Native Matrix needs no bridge and groups the same way: each room gets one message per sweep (a lone alert keeps the full single-alert message). Neither is held by the time-window digest, because both carry a secret (headers or an access token) that is never written to the buffer. If a grouped send fails, its alerts are retried on the next sweep.

A custom webhook target with the slack format is grouped the same way: one sweep sends one Slack message with a summary line, the findings listed (capped at 25 with "…and N more", and kept inside Slack's text limits), a colour bar from the highest severity (green when every finding is a recovery), and an "Open dashboard" button. A lone finding keeps the full single-alert message. A target with a title or body template still sends one message per finding.

Scheduling (Cloudflare Cron)

The hosted deploy schedules the sweep every 10 minutes. In wrangler.toml:

[triggers]
crons = ["0 3 * * *", "0 8 * * 1", "*/10 * * * *"]  # retention, weekly-report, health-sweep

The Cloudflare scheduled trigger is routed to the route's GET handler. Two gates apply, in order:

  1. Auth — the handler requires CRON_SECRET (returns HTTP 503 when unset). Configure the secret and invoke with Authorization: Bearer <secret> only — e.g. from an external scheduler, or a Worker Cron that forwards the secret. Query-string ?secret= is rejected.
  2. Enablement — after auth, the handler consults CHM_HEALTH_SWEEP_ENABLED. When it is falsy the route returns 200 {"skipped": true} without running the sweep, so you can pause scheduled alerting without touching the cron or the secret. When unset it defaults to enabled (since CRON_SECRET is already present at that point).

The */10 cadence (rather than */5) leaves CPU-time headroom: the sweep also generates AI insights per host, so its cost scales with the number of configured CLICKHOUSE_HOST entries. For very large host fleets, either lengthen the cadence or pause the scheduled sweep (CHM_HEALTH_SWEEP_ENABLED=false) and drive it from an external scheduler.

Weekly health report

Beyond the health-sweep, a second cron job builds a proactive weekly HTML report per opted-in host: a 7-day narrative combining the AI insights engine's top findings, the number of adaptive statistical baselines fitted to that cluster, and a disk-capacity outlook from the capacity forecaster.

Each report also includes real cluster data over the window, when the host's system tables allow it (each section is fail-open and simply omitted otherwise):

  • Query activity — total and failed query counts, p50/p95 duration, and a per-day sparkline (from system.query_log).
  • Ingestion — rows and bytes written by INSERTs per day, with sparklines (from system.query_log; uncompressed measurement, same model as the Traffic page).
  • Storage — total on-disk size plus the top 5 tables by size, each annotated with the bytes written during the window (from active system.parts).

Reports are opt-in, never opt-out — set CHM_WEEKLY_REPORT_HOSTS to a comma-separated list of host indices (e.g. 0,2) to enable it for those hosts. Leave it unset and no reports are generated, so a self-hosted deployment stays quiet by default.

VariableDefaultDescription
CHM_WEEKLY_REPORT_HOSTSunset (no reports)Comma-separated host indices to opt in, e.g. 0,2. Garbage entries are dropped; an opted-in index with no matching configured host is skipped with a warning.

The report is generated by GET /api/cron/weekly-report, guarded by the same CRON_SECRET as the health-sweep endpoint above, and scheduled weekly on Mondays at 08:00 UTC:

[triggers]
crons = ["0 8 * * 1"]

The report is always persisted, even when no delivery channel is configured — so a deployment with no webhook still accrues viewable reports. When HEALTH_ALERT_WEBHOOK_URL is set, the report is also best-effort posted to that same webhook.

View or download the most recent persisted report:

# Self-contained HTML document
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0"

# Parsed JSON summary instead
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=json"

# A specific past week (YYYY-MM-DD week-start)
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&week=2026-07-06"

# PDF (Cloudflare Browser Rendering; falls back to HTML when unavailable)
curl -L "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=pdf" -o report.pdf

Scheduled reports (per-user)

Beyond the env-var opt-in above, Scheduled Reports (/report-settings in the dashboard) lets each user subscribe to a weekly or monthly cluster health report delivered through their already-configured alert channels — email (full HTML), webhook, Telegram, ntfy, and Pushover. Paging channels (Opsgenie, PagerDuty, Twilio) are deliberately excluded: a digest must never page anyone.

  • Cadence — monthly (1st of the month, 30-day window, GET /api/cron/monthly-report, cron 0 8 1 * *) or weekly (Mondays, 7-day window, shares the weekly cron above). On chmonitor Cloud, weekly cadence requires a paid plan; monthly is included on Free.
  • Generate now — build a fresh report on demand and open/download the self-contained HTML. Works on any deployment with zero delivery configuration (POST /api/v1/reports/generate).
  • Download PDF — render the same report to PDF via Cloudflare Browser Rendering (POST /api/v1/reports/generate with { "format": "pdf" }, or ?format=pdf on the read API). PDF export is a Pro+ (data_export) capability on chmonitor Cloud. It is optional and fail-closed: when no BROWSER binding is configured (self-hosted Docker / K8s, or any Cloudflare account without Browser Rendering) or a render fails, the request degrades gracefully to HTML (signalled by an X-Report-PDF: unavailable header). On Cloud, a Pro+ subscriber's scheduled email delivery also gets the PDF attached automatically.
  • Send test report — deliver one immediately through your configured channels and see the per-channel outcome (POST /api/v1/reports/test-send).
  • Audit — the subscription stores the last delivery time and a compact per-channel status.
  • Fleet report — a subscription covering more than one host is delivered as ONE combined fleet report instead of N separate ones: a comparison table (findings, critical, queries, ingested bytes, disk %) followed by each host's full sections. Per-host reports are still persisted individually.

Subscriptions live in the report_subscriptions D1 table (owner-scoped, like alert channels); deployments without D1 keep the env-var webhook flow and the on-demand Generate button. The AI agent can also produce the same report conversationally via its generate_cluster_report tool.

Notes & limitations

  • system.error_log is checked separately by the Errors page (under Operations). The Health page uses system.errors (in-memory error counts), which resets on server restart.
  • If a system table is missing (e.g., no system.replicas on a standalone node), that check is skipped with an "unavailable" state rather than a false positive.
  • The webhook payload is a plain Slack/Discord-compatible JSON object ({"text": "...", "content": "..."}). Custom payload shapes are not supported in v1.
  • The health-sweep endpoint queries all configured CLICKHOUSE_HOST entries. High host counts increase sweep latency.
  • If PeerDB is configured (PEERDB_API_URL), the sweep also checks every PeerDB mirror and alerts as peerdb:<mirror> through the same channels and routes. Set CHM_FEATURE_PEERDB_ENABLED=false to skip it. See PeerDB Monitoring.

On this page