Health

Get a single status grid of cluster health across replication, merges, errors, disks, load, and parts — plus a headless sweep endpoint that fires webhook alerts on a schedule.

What it does

The Health page aggregates checks across multiple system tables into a single status grid. Each check has a severity level (ok / warning / critical) and a human-readable summary. Operators use this page as a starting point for incident triage.

Checks cover:

• Replication lag — replicas falling behind, readonly tables
• Merge backlog — active merge count and slow merges
• Error rate — recent errors from system.errors
• Disk usage — free space across all disks
• Query load — running query count and memory pressure
• Part health — excessive part counts
• TTL & Partition Health — MergeTree tables with partition bloat, a time-based PARTITION BY and no table TTL, or a merge backlog (recommend-only; full inventory on /ttl-partition-health)

In addition to the UI, chmonitor exposes a headless health-sweep endpoint (GET /api/cron/health-sweep) that runs the same checks and dispatches webhook alerts. This is designed to be called on a schedule (e.g., Cloudflare Cron every 5 minutes) without a browser.

Pages

Using it

Open /health for the status grid — each check reports ok / warning / critical with a summary. Start incident triage here, then drill into the linked feature sections for detail.

To run the same checks without a browser, call the health-sweep endpoint (see below) on a schedule.

Alert Settings layout

/alert-settings has four tabs: Alerts, Thresholds, Activity, and Advanced. The Alerts tab answers "what alerts do I have, and where do they go?" on one screen:

1. Built-in alerts — one row per health check, with its name and stable id.
2. More ways to alert — Suggested alerts (thresholds derived from how this cluster behaved) and Custom rules (your own rule on any metric).
3. Channels — browser, server delivery channels, and custom webhook targets.
4. Delivery — Routing rules (which alerts go to which channel), Webhook subscriptions (POST alert events to your own endpoints), and Digest (batch non-critical alerts).
5. Minimum severity — the default floor every channel inherits.

The Advanced tab now holds only Silencing: Quiet hours and Maintenance windows. Each item still opens in its own dialog. Old links keep working: ?tab=routing, ?tab=webhooks, ?tab=digest, ?tab=suggested, and ?tab=custom-rules open that dialog on the Alerts tab; ?tab=maintenance and ?tab=quiet-hours open it on Advanced. ?tab=define, ?tab=delivery, and ?tab=silencing open the tab that holds that group.

Naming built-in alerts

Each built-in alert can be renamed from the list (1–80 characters). The name is display-only in the dashboard: notifications still use the built-in check title, and the alert keeps its check id, so a rename never resets alert state or acknowledgements. Names are stored in the metadata database; with no database the list still renders, but rename is read-only.

Permissions & access

CHM_FEATURE_HEALTH_ACCESS=authenticated

CHM_FEATURE_HEALTH_ENABLED=false

[features.health]
enabled = true
access = "authenticated"

Configuration

Run alerts without a metadata database

Alerting works on Docker and Kubernetes with no D1 and no Postgres, but there is one boundary to learn up front: declarative config can fire alerts. It cannot acknowledge them.

With no database, alert state lives in memory in each worker instance. So hysteresis streaks and incident timers reset on every restart or deploy, and Acknowledge, Clear and dismiss are disabled in the UI. POST and DELETE on /api/v1/health/ack return 501. The ClickHouse state backend (CHM_STATE_CLICKHOUSE_*) does not back the alert stores; the alert surface runs on D1 or Postgres (DATABASE_URL / POSTGRES_URL). Attach one of them if you need ACKs. See GET /api/v1/config capabilities.health for what a deployment resolved.

In-memory alert state is not migrated when you attach a database later. On the first sweep that reaches the new database, the in-memory state is discarded and alerting starts fresh from the database: a pending hysteresis streak restarts, and a condition that is still firing opens a new incident and can notify again.

Where definitions come from

Definitions merge from four sources, highest first:

1. D1 or Postgres rows, written from the UI.
2. Config files in CHM_HEALTH_CONFIG_DIRECTORY (default /etc/chmonitor/health.d).
3. Environment variables (HEALTH_ALERT_*, HEALTH_THRESHOLD_*).
4. Built-in defaults.

Every source is always read; precedence only decides who wins a key. Per key:

• Most definitions (routes, custom rules, webhook targets, quiet hours, channels) merge field by field. A higher source overrides a field only when it sets a non-empty value, so a file can set enabled while a D1 row owns a token. A higher source cannot clear a lower source's value back to empty.
• Maintenance windows are replaced whole per id, and windows from different sources are all active.
• Digest settings are a single value: the highest source wins outright.
• A key that only a lower source defines disappears when that source stops defining it. There is no tombstone.

Declared entries are shared by every user (deployment-wide, like the env fallbacks). In the UI they are read-only and show a Config file or Env badge instead of Delete or Reset. Custom rules declared with id: foo appear as custom:foo. The directory is read once per process, so a changed file needs a pod restart. Secrets are read from the environment on each read.

The files

One YAML file per concern, named exactly <concern>.yaml (or .yml). Files with other names are skipped with a warning.

Every keyed entry carries its key explicitly. The schema is strict: unknown keys are rejected, so a typo is reported instead of silently ignored.

Secrets never go in these files. A ConfigMap is not a Secret. Secret fields take the name of an environment variable (secretEnv, urlEnv, headersEnv), which you back with a Kubernetes Secret. An unset variable skips that entry with a warning that names the variable, never its value. Declared outbound URLs must be HTTPS and pass the same SSRF check as the API. A URL that is itself a credential (Slack or Discord webhook) belongs in a routes entry secretEnv or a webhookTargets urlEnv, not in a channel target.url.

alerts.yaml declares custom rules. metric is a key from the rule builder catalog (for example failed-mutations, readonly-replicas, replication-max-lag, disk-usage-percent). op is one of >, >=, <, <=, and warning and critical must be ordered for that operator.

rules:
  - id: failed-mutations-any
    name: Failed mutations
    metric: failed-mutations
    op: ">="
    warning: 1
    critical: 5

alerts.yaml also accepts a thresholds: map (rule id to warning / critical). The sweep and the current-findings snapshot apply it over the built-in default. It is merged with the HEALTH_THRESHOLD_<RULE>_WARNING and HEALTH_THRESHOLD_<RULE>_CRITICAL environment variables field by field, and the file wins a collision: warning: 70 in the file with HEALTH_THRESHOLD_DISK_USAGE_CRITICAL=95 in env gives 70 / 95. A rule id that is not registered is skipped with a warning in the log. The file is read once per process, so restart after editing it; env changes apply on the next sweep. Only built-in rule ids are honoured by the current-findings snapshot.

thresholds:
  disk-usage:
    warning: 70

routing.yaml declares routes. provider is webhook, pagerduty, telegram, ntfy or pushover; matchRule and matchHost default to * and accept globs. secretEnv names the variable holding the webhook URL, PagerDuty routing key, Telegram bot token, ntfy token (optional) or Pushover app token. target carries the non-secret fields: serviceName (pagerduty), chatId (telegram), url (ntfy) and user (pushover).

routes:
  - id: oncall-pagerduty
    provider: pagerduty
    matchRule: "*"
    matchHost: "*"
    minSeverity: critical
    secretEnv: CHM_ROUTE_PAGERDUTY_KEY
    target:
      serviceName: chmonitor-prod
  - id: team-telegram
    provider: telegram
    matchRule: "custom:*"
    secretEnv: CHM_ROUTE_TELEGRAM_TOKEN
    target:
      chatId: "-1001234567890"

channels.yaml declares the server delivery channels (webhook, healthchecks, email, opsgenie, telegram, ntfy, pushover, twilio) and custom webhook targets. A target's urlEnv is required; headersEnv names a variable holding a JSON object of secret headers. Precedence against env is per channel, not per field.

channels:
  - channel: telegram
    minSeverity: warning
    secretEnv: HEALTH_CHANNEL_TELEGRAM_TOKEN
    target:
      chatId: "-1001234567890"
  - channel: opsgenie
    secretEnv: HEALTH_CHANNEL_OPSGENIE_KEY
    target:
      region: eu
webhookTargets:
  - id: ops-chat
    name: Ops chat
    format: slack
    minSeverity: warning
    urlEnv: CHM_WEBHOOK_OPS_CHAT_URL
    headersEnv: CHM_WEBHOOK_OPS_CHAT_HEADERS

quiet-hours.yaml takes days (0 to 6), start and end as HH:mm, an IANA timezone (default UTC) and an optional severityCap: critical.

windows:
  - id: nightly
    days: [1, 2, 3, 4, 5]
    start: "22:00"
    end: "07:00"
    timezone: Asia/Ho_Chi_Minh
    severityCap: critical

maintenance.yaml takes an optional hostId (omit for all hosts), a reason, and ISO-8601 startsAt / endsAt with endsAt later.

windows:
  - id: upgrade-2026-10-05
    hostId: 0
    reason: ClickHouse 25.8 upgrade
    startsAt: "2026-10-05T01:00:00Z"
    endsAt: "2026-10-05T03:00:00Z"

digest.yaml is a single object. windowMinutes is 0 to 1440.

enabled: true
windowMinutes: 15

Fail-closed behaviour

• A missing directory, or a runtime with no filesystem (Cloudflare Workers), is a silent no-op.
• Bad YAML, unknown keys, schema violations, entries without a key and duplicate keys are skipped with a warning. The rest of the file and the other files still load, and the pod never fails to start.
• Warnings name the file and entry, never a field value, so a secret pasted into a ConfigMap by mistake does not reach the logs.
• An entry whose *Env variable is unset, or whose URL is not an allowed HTTPS endpoint, is skipped.
• With no database and no declared custom rule, the rule builder still reports "not available" (501) on write.

For the Kubernetes mount, see Alert definitions from a ConfigMap. Variable reference: CHM_HEALTH_CONFIG_DIRECTORY.

Health-sweep cron alerting

The health-sweep endpoint runs checks over all configured hosts and sends a webhook notification when a check meets or exceeds the minimum severity.

wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URL
# Optional: Telegram channel (bot token is a secret; chat id is a plain var)
wrangler secret put HEALTH_ALERT_TELEGRAM_BOT_TOKEN
# Optional: ntfy channel (topic URL is a plain var; access token is a secret)
wrangler secret put HEALTH_ALERT_NTFY_TOKEN
# Optional: Twilio SMS channel (auth token is a secret; SID/from/to are plain vars)
wrangler secret put HEALTH_ALERT_TWILIO_AUTH_TOKEN
# Optional: Pushover channel (app token is a secret; user key is a plain var)
wrangler secret put HEALTH_ALERT_PUSHOVER_TOKEN

HEALTH_ALERT_ENABLED=true
HEALTH_ALERT_MIN_SEVERITY=warning
HEALTH_ALERT_WEBHOOK_URL=https://hooks.slack.com/services/...
CRON_SECRET=<random-secret>

curl -H "Authorization: Bearer $CRON_SECRET" \
  https://your-chmonitor.example.com/api/cron/health-sweep

Scheduling (Cloudflare Cron)

The hosted deploy schedules the sweep every 10 minutes. In wrangler.toml:

[triggers]
crons = ["0 3 * * *", "0 8 * * 1", "*/10 * * * *"]  # retention, weekly-report, health-sweep

The Cloudflare scheduled trigger is routed to the route's GET handler. Two gates apply, in order:

1. Auth — the handler requires CRON_SECRET (returns HTTP 503 when unset). Configure the secret and invoke with Authorization: Bearer <secret> only — e.g. from an external scheduler, or a Worker Cron that forwards the secret. Query-string ?secret= is rejected.
2. Enablement — after auth, the handler consults CHM_HEALTH_SWEEP_ENABLED. When it is falsy the route returns 200 {"skipped": true} without running the sweep, so you can pause scheduled alerting without touching the cron or the secret. When unset it defaults to enabled (since CRON_SECRET is already present at that point).

The */10 cadence (rather than */5) leaves CPU-time headroom: the sweep also generates AI insights per host, so its cost scales with the number of configured CLICKHOUSE_HOST entries. For very large host fleets, either lengthen the cadence or pause the scheduled sweep (CHM_HEALTH_SWEEP_ENABLED=false) and drive it from an external scheduler.

Weekly health report

Beyond the health-sweep, a second cron job builds a proactive weekly HTML report per opted-in host: a 7-day narrative combining the AI insights engine's top findings, the number of adaptive statistical baselines fitted to that cluster, and a disk-capacity outlook from the capacity forecaster.

Each report also includes real cluster data over the window, when the host's system tables allow it (each section is fail-open and simply omitted otherwise):

• Query activity — total and failed query counts, p50/p95 duration, and a per-day sparkline (from system.query_log).
• Ingestion — rows and bytes written by INSERTs per day, with sparklines (from system.query_log; uncompressed measurement, same model as the Traffic page).
• Storage — total on-disk size plus the top 5 tables by size, each annotated with the bytes written during the window (from active system.parts).

Reports are opt-in, never opt-out — set CHM_WEEKLY_REPORT_HOSTS to a comma-separated list of host indices (e.g. 0,2) to enable it for those hosts. Leave it unset and no reports are generated, so a self-hosted deployment stays quiet by default.

The report is generated by GET /api/cron/weekly-report, guarded by the same CRON_SECRET as the health-sweep endpoint above, and scheduled weekly on Mondays at 08:00 UTC:

[triggers]
crons = ["0 8 * * 1"]

The report is always persisted, even when no delivery channel is configured — so a deployment with no webhook still accrues viewable reports. When HEALTH_ALERT_WEBHOOK_URL is set, the report is also best-effort posted to that same webhook.

View or download the most recent persisted report:

# Self-contained HTML document
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0"

# Parsed JSON summary instead
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=json"

# A specific past week (YYYY-MM-DD week-start)
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&week=2026-07-06"

# PDF (Cloudflare Browser Rendering; falls back to HTML when unavailable)
curl -L "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=pdf" -o report.pdf

Scheduled reports (per-user)

Beyond the env-var opt-in above, Scheduled Reports (/report-settings in the dashboard) lets each user subscribe to a weekly or monthly cluster health report delivered through their already-configured alert channels — email (full HTML), webhook, Telegram, ntfy, and Pushover. Paging channels (Opsgenie, PagerDuty, Twilio) are deliberately excluded: a digest must never page anyone.

• Cadence — monthly (1st of the month, 30-day window, GET /api/cron/monthly-report, cron 0 8 1 * *) or weekly (Mondays, 7-day window, shares the weekly cron above). On chmonitor Cloud, weekly cadence requires a paid plan; monthly is included on Free.
• Generate now — build a fresh report on demand and open/download the self-contained HTML. Works on any deployment with zero delivery configuration (POST /api/v1/reports/generate).
• Download PDF — render the same report to PDF via Cloudflare Browser Rendering (POST /api/v1/reports/generate with { "format": "pdf" }, or ?format=pdf on the read API). PDF export is a Pro+ (data_export) capability on chmonitor Cloud. It is optional and fail-closed: when no BROWSER binding is configured (self-hosted Docker / K8s, or any Cloudflare account without Browser Rendering) or a render fails, the request degrades gracefully to HTML (signalled by an X-Report-PDF: unavailable header). On Cloud, a Pro+ subscriber's scheduled email delivery also gets the PDF attached automatically.
• Send test report — deliver one immediately through your configured channels and see the per-channel outcome (POST /api/v1/reports/test-send).
• Audit — the subscription stores the last delivery time and a compact per-channel status.
• Fleet report — a subscription covering more than one host is delivered as ONE combined fleet report instead of N separate ones: a comparison table (findings, critical, queries, ingested bytes, disk %) followed by each host's full sections. Per-host reports are still persisted individually.

Subscriptions live in the report_subscriptions D1 table (owner-scoped, like alert channels); deployments without D1 keep the env-var webhook flow and the on-demand Generate button. The AI agent can also produce the same report conversationally via its generate_cluster_report tool.

Notes & limitations

• system.error_log is checked separately by the Errors page (under Operations). The Health page uses system.errors (in-memory error counts), which resets on server restart.
• If a system table is missing (e.g., no system.replicas on a standalone node), that check is skipped with an "unavailable" state rather than a false positive.
• The webhook payload is a plain Slack/Discord-compatible JSON object ({"text": "...", "content": "..."}). Custom payload shapes are not supported in v1.
• The health-sweep endpoint queries all configured CLICKHOUSE_HOST entries. High host counts increase sweep latency.

Related