Health

Get a single status grid of cluster health across replication, merges, errors, disks, load, and parts — plus a headless sweep endpoint that fires webhook alerts on a schedule.

What it does

The Health page aggregates checks across multiple system tables into a single status grid. Each check has a severity level (ok / warning / critical) and a human-readable summary. Operators use this page as a starting point for incident triage.

Checks cover:

• Replication lag — replicas falling behind, readonly tables
• Merge backlog — active merge count and slow merges
• Error rate — recent errors from system.errors
• Disk usage — free space across all disks
• Query load — running query count and memory pressure
• Part health — excessive part counts

In addition to the UI, chmonitor exposes a headless health-sweep endpoint (GET /api/cron/health-sweep) that runs the same checks and dispatches webhook alerts. This is designed to be called on a schedule (e.g., Cloudflare Cron every 5 minutes) without a browser.

Pages

Using it

Open /health for the status grid — each check reports ok / warning / critical with a summary. Start incident triage here, then drill into the linked feature sections for detail.

To run the same checks without a browser, call the health-sweep endpoint (see below) on a schedule.

Permissions & access

CHM_FEATURE_HEALTH_ACCESS=authenticated

CHM_FEATURE_HEALTH_ENABLED=false

[features.health]
enabled = true
access = "authenticated"

Configuration

Health-sweep cron alerting

The health-sweep endpoint runs checks over all configured hosts and sends a webhook notification when a check meets or exceeds the minimum severity.

wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URL
# Optional: Telegram channel (bot token is a secret; chat id is a plain var)
wrangler secret put HEALTH_ALERT_TELEGRAM_BOT_TOKEN
# Optional: ntfy channel (topic URL is a plain var; access token is a secret)
wrangler secret put HEALTH_ALERT_NTFY_TOKEN
# Optional: Twilio SMS channel (auth token is a secret; SID/from/to are plain vars)
wrangler secret put HEALTH_ALERT_TWILIO_AUTH_TOKEN
# Optional: Pushover channel (app token is a secret; user key is a plain var)
wrangler secret put HEALTH_ALERT_PUSHOVER_TOKEN

HEALTH_ALERT_ENABLED=true
HEALTH_ALERT_MIN_SEVERITY=warning
HEALTH_ALERT_WEBHOOK_URL=https://hooks.slack.com/services/...
CRON_SECRET=<random-secret>

curl -H "Authorization: Bearer $CRON_SECRET" \
  https://your-chmonitor.example.com/api/cron/health-sweep

Scheduling (Cloudflare Cron)

The hosted deploy schedules the sweep every 10 minutes. In wrangler.toml:

[triggers]
crons = ["0 3 * * *", "0 8 * * 1", "*/10 * * * *"]  # retention, weekly-report, health-sweep

The Cloudflare scheduled trigger is routed to the route's GET handler. Two gates apply, in order:

1. Auth — the handler requires CRON_SECRET (returns HTTP 503 when unset). Configure the secret and invoke with Authorization: Bearer <secret> (or ?secret=<secret>) — e.g. from an external scheduler, or a Worker Cron that forwards the secret. An unauthenticated scheduled hit is rejected by design.
2. Enablement — after auth, the handler consults CHM_HEALTH_SWEEP_ENABLED. When it is falsy the route returns 200 {"skipped": true} without running the sweep, so you can pause scheduled alerting without touching the cron or the secret. When unset it defaults to enabled (since CRON_SECRET is already present at that point).

The */10 cadence (rather than */5) leaves CPU-time headroom: the sweep also generates AI insights per host, so its cost scales with the number of configured CLICKHOUSE_HOST entries. For very large host fleets, either lengthen the cadence or pause the scheduled sweep (CHM_HEALTH_SWEEP_ENABLED=false) and drive it from an external scheduler.

Weekly health report

Beyond the health-sweep, a second cron job builds a proactive weekly HTML report per opted-in host: a 7-day narrative combining the AI insights engine's top findings, the number of adaptive statistical baselines fitted to that cluster, and a disk-capacity outlook from the capacity forecaster.

Each report also includes real cluster data over the window, when the host's system tables allow it (each section is fail-open and simply omitted otherwise):

• Query activity — total and failed query counts, p50/p95 duration, and a per-day sparkline (from system.query_log).
• Ingestion — rows and bytes written by INSERTs per day, with sparklines (from system.query_log; uncompressed measurement, same model as the Traffic page).
• Storage — total on-disk size plus the top 5 tables by size, each annotated with the bytes written during the window (from active system.parts).

Reports are opt-in, never opt-out — set CHM_WEEKLY_REPORT_HOSTS to a comma-separated list of host indices (e.g. 0,2) to enable it for those hosts. Leave it unset and no reports are generated, so a self-hosted deployment stays quiet by default.

The report is generated by GET /api/cron/weekly-report, guarded by the same CRON_SECRET as the health-sweep endpoint above, and scheduled weekly on Mondays at 08:00 UTC:

[triggers]
crons = ["0 8 * * 1"]

The report is always persisted, even when no delivery channel is configured — so a deployment with no webhook still accrues viewable reports. When HEALTH_ALERT_WEBHOOK_URL is set, the report is also best-effort posted to that same webhook.

View or download the most recent persisted report:

# Self-contained HTML document
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0"

# Parsed JSON summary instead
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=json"

# A specific past week (YYYY-MM-DD week-start)
curl "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&week=2026-07-06"

# PDF (Cloudflare Browser Rendering; falls back to HTML when unavailable)
curl -L "https://your-chmonitor.example.com/api/v1/insights/weekly-report?host=0&format=pdf" -o report.pdf

Scheduled reports (per-user)

Beyond the env-var opt-in above, Scheduled Reports (/report-settings in the dashboard) lets each user subscribe to a weekly or monthly cluster health report delivered through their already-configured alert channels — email (full HTML), webhook, Telegram, ntfy, and Pushover. Paging channels (Opsgenie, PagerDuty, Twilio) are deliberately excluded: a digest must never page anyone.

• Cadence — monthly (1st of the month, 30-day window, GET /api/cron/monthly-report, cron 0 8 1 * *) or weekly (Mondays, 7-day window, shares the weekly cron above). On chmonitor Cloud, weekly cadence requires a paid plan; monthly is included on Free.
• Generate now — build a fresh report on demand and open/download the self-contained HTML. Works on any deployment with zero delivery configuration (POST /api/v1/reports/generate).
• Download PDF — render the same report to PDF via Cloudflare Browser Rendering (POST /api/v1/reports/generate with { "format": "pdf" }, or ?format=pdf on the read API). PDF export is a Pro+ (data_export) capability on chmonitor Cloud. It is optional and fail-closed: when no BROWSER binding is configured (self-hosted Docker / K8s, or any Cloudflare account without Browser Rendering) or a render fails, the request degrades gracefully to HTML (signalled by an X-Report-PDF: unavailable header). On Cloud, a Pro+ subscriber's scheduled email delivery also gets the PDF attached automatically.
• Send test report — deliver one immediately through your configured channels and see the per-channel outcome (POST /api/v1/reports/test-send).
• Audit — the subscription stores the last delivery time and a compact per-channel status.
• Fleet report — a subscription covering more than one host is delivered as ONE combined fleet report instead of N separate ones: a comparison table (findings, critical, queries, ingested bytes, disk %) followed by each host's full sections. Per-host reports are still persisted individually.

Subscriptions live in the report_subscriptions D1 table (owner-scoped, like alert channels); deployments without D1 keep the env-var webhook flow and the on-demand Generate button. The AI agent can also produce the same report conversationally via its generate_cluster_report tool.

Notes & limitations

• system.error_log is checked separately by the Errors page (under Operations). The Health page uses system.errors (in-memory error counts), which resets on server restart.
• If a system table is missing (e.g., no system.replicas on a standalone node), that check is skipped with an "unavailable" state rather than a false positive.
• The webhook payload is a plain Slack/Discord-compatible JSON object ({"text": "...", "content": "..."}). Custom payload shapes are not supported in v1.
• The health-sweep endpoint queries all configured CLICKHOUSE_HOST entries. High host counts increase sweep latency.

Related