Alerting to Slack and Discord

Wire chmonitor's health sweep to a Slack or Discord incoming webhook so your team gets notified the moment a check crosses a threshold — no polling the dashboard required.

chmonitor's Health page checks replication lag, failed mutations, disk usage and a handful of other signals on a schedule you control. Left alone, that only helps the person who remembers to look at the page. The health-sweep cron endpoint closes that gap: point it at a Slack or Discord incoming webhook and it posts a message whenever a check meets or exceeds a severity threshold.

This page walks through creating the webhook URL itself. For the full CRON_SECRET / HEALTH_ALERT_* environment variable reference, scheduling on Cloudflare Cron, and the sequence diagram, see Health → Health-sweep cron alerting.

Prerequisites

A chmonitor instance already connected to at least one ClickHouse host.
The ability to set environment variables / secrets on your deployment (Cloudflare Worker secret, Docker env, or Kubernetes Secret).

1. Create the webhook URL

Go to api.slack.com/apps → create (or open) an app → Incoming Webhooks → turn the toggle on.
Add New Webhook to Workspace, pick the channel, and copy the generated URL (https://hooks.slack.com/services/...).

In the target channel: Channel Settings → Integrations → Webhooks → New Webhook.
Name it, optionally set an avatar, and Copy Webhook URL.

The same URL works for both — the sweep posts {"text": "...", "content": "..."}, and each platform reads the field it understands.

2. Configure and enable

CRON_SECRET=<random-secret>
HEALTH_ALERT_ENABLED=true
HEALTH_ALERT_MIN_SEVERITY=warning   # or "critical" to only page on the worst checks
HEALTH_ALERT_WEBHOOK_URL=<the URL you copied above>

On Cloudflare Workers, set the secrets rather than plain vars:

wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URL

3. Test it

curl -H "Authorization: Bearer $CRON_SECRET" \
  https://your-chmonitor.example.com/api/cron/health-sweep

The endpoint always returns HTTP 200 with a JSON array of check results — dispatch happens server-side and doesn't block the response. To confirm a message actually went out, trip a threshold intentionally (temporarily lower HEALTH_ALERT_MIN_SEVERITY to warning) and watch the channel.

Alert history

On Cloud with D1 configured, the Health page's alert history records every dispatch attempt and whether delivery succeeded. That history isn't available on a self-hosted deployment without D1.

Noise control: transitions, recovery & hysteresis

Alerts are transition-based: the sweep notifies only when a condition actually changes state — ok → warning, warning → critical, or recovers back to ok. A condition that stays unhealthy is not re-sent on every run; it sends a single reminder at most once per HEALTH_ALERT_COOLDOWN_MINUTES (default 60; 0 disables reminders). A recovery message always goes out when a condition clears, and reports how long the incident was firing.

To stop a metric that hovers on a threshold from flapping (fire/recover on every sweep), each check applies hysteresis:

HEALTH_HYSTERESIS_BREACHES=1
# Consecutive "ok" sweeps required before a firing condition recovers (default 2).
HEALTH_HYSTERESIS_CLEARS=2

# Per-check overrides use the rule id, uppercased with dashes → underscores:
HEALTH_HYSTERESIS_DISK_USAGE_CLEARS=3

The default (fire on the first breach, recover only after two consecutive clears) keeps genuine criticals prompt while collapsing a flapping metric into exactly one fire and one recover. The current state and last transition per check are shown on the Active tab of Alert Settings. State is persisted to D1 (when configured) so hysteresis streaks and incident timers survive worker restarts.

Health — every check, threshold, and environment variable, plus scheduling on Cloudflare Cron.
Slack app — the native /chmonitor Slack app (slash command, Home tab, Acknowledge buttons), a different and more capable integration than a plain incoming webhook.