Guide
Alerting to Slack and Discord
Wire chmonitor's health sweep to a Slack or Discord incoming webhook so your team gets notified the moment a check crosses a threshold — no polling the dashboard required.
chmonitor's Health page checks replication lag, failed mutations, disk usage and a handful of other signals on a schedule you control. Left alone, that only helps the person who remembers to look at the page. The health-sweep cron endpoint closes that gap: point it at a Slack or Discord incoming webhook and it posts a message whenever a check meets or exceeds a severity threshold.
This page walks through creating the webhook URL itself. For the full
CRON_SECRET / HEALTH_ALERT_* environment variable reference, scheduling on
Cloudflare Cron, and the sequence diagram, see
Health → Health-sweep cron alerting.
Prerequisites
- A chmonitor instance already connected to at least one ClickHouse host.
- The ability to set environment variables / secrets on your deployment (Cloudflare Worker secret, Docker env, or Kubernetes Secret).
1. Create the webhook URL
- Go to api.slack.com/apps → create (or open) an app → Incoming Webhooks → turn the toggle on.
- Add New Webhook to Workspace, pick the channel, and copy the generated
URL (
https://hooks.slack.com/services/...).
- In the target channel: Channel Settings → Integrations → Webhooks → New Webhook.
- Name it, optionally set an avatar, and Copy Webhook URL.
The same URL works for both — the sweep posts {"text": "...", "content": "..."},
and each platform reads the field it understands.
2. Configure and enable
CRON_SECRET=<random-secret>
HEALTH_ALERT_MIN_SEVERITY=warning # or "critical" to only page on the worst checks
HEALTH_ALERT_WEBHOOK_URL=<the URL you copied above>On Cloudflare Workers, set the secrets rather than plain vars:
wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URLHEALTH_ALERT_ENABLED is optional. Alerting defaults to on as soon as
HEALTH_ALERT_WEBHOOK_URL is set; set HEALTH_ALERT_ENABLED=false to pause
alerts without removing the URL. The same rule applies to email
(HEALTH_ALERT_EMAIL_ENABLED): it is on once the provider URL, from and to
addresses are set.
Custom formats and multiple targets
For more control, open Health settings → Alerts → Custom webhook targets. Each target can select:
- Raw JSON — the normalized alert payload for a generic receiver.
- Slack — escaped Slack blocks.
- Element / Matrix — an
m.noticeevent with escaped HTML. - Auto — preserve the legacy URL-based format detection.
- Matrix Hookshot — the
{text, html}body that matrix-hookshot generic webhooks expect. Use this, not Element / Matrix, for a Hookshot URL: Hookshot posts any body without atextkey to the room as raw JSON.
Matrix Hookshot and Element / Matrix targets with no title or body
template are grouped per sweep: when one sweep finds several alerts for the
same target, the room gets ONE message. It starts with a coloured summary line
(3 critical, 1 warning on 4 hosts), lists the findings under each rule
(host, metric = value, severity), stops after 25 lines with …and N more, and
ends with one dashboard link. A single finding is sent exactly as before.
Grouped targets are sent in the same sweep even when the time-window digest is
on; they are never written to the digest buffer, because they can carry secret
headers. A target with a template sends one message per finding.
A Slack format target sends one compact line per alert next to a severity
colour bar, for example
🔴 CRITICAL Stuck Merges · node-2 · 4 (warn 1, crit 3) · Open. A recovery
reads 🟢 RESOLVED … · fired for 35 min · Open. The instance prefix appears
only when instanceName is set; runbook links and metadata go on one small
second line when there are any. A grouped Slack digest is compact text: a
bold summary line, one line per rule in the same layout as a single alert, and
a small gray footer. Several alerts of one rule and severity share one line:
the rule title with a count, up to three names then +N, and the age of the
oldest alert, for example 🟠 WARNING Source stopped sending rows (34) · db.events:source-1, db.events:source-2, db.events:source-3, +31 · 3 h. A lone
alert shows its full timeline (still open, 3 h (88.5 -> 97.25), fired for 35 min), or new. Recoveries get their own line. The summary line counts every
alert. When the lines share one host, the summary names it and the lines leave
it out; otherwise each line adds the hosts. Past 25 lines the digest ends with
…and N more. The footer holds the Open dashboard link, up to three
runbook links, the instance name and metadata, and the sweep time in UTC. A sweep that only adds alerts of a rule that is already open, or only
resolves, posts one line instead of the digest: +1 warning · 1 resolved · 60 open.
The form supports bounded {{severity}}, {{title}}, {{label}}, {{host}},
{{value}}, and {{timestamp}} templates plus safe X-* headers. Preview
uses the same server-side formatter as delivery and masks the destination URL;
Send test is an explicit write-gated request. Target URLs are credentials:
they are never returned to the browser, and authorization headers belong in a
Kubernetes Secret or another server-side secret source.
On Kubernetes, the Helm chart can declare the same targets under
alertWebhooks.targets for GitOps-managed deployments. Use urlFrom and
headersSecretFrom with an existing Secret; do not commit token-bearing URLs in
values.yaml, a ConfigMap, or extraEnv. D1/UI configuration is owner-scoped
and takes precedence over a Helm target with the same name; deleting the D1 row
returns to the Helm declaration.
Structured alert block for agents
Set details: true on a Raw JSON custom target, or on a webhook
subscription ("details": true in POST/PATCH /api/v1/webhooks/subscriptions), to add one alert block to the body. Chat
receivers and agents read the same schema. Off (the default), the body is
unchanged. Subscriptions carry it as data.alert on alert.fired and
alert.resolved.
{
"alert": {
"ruleId": "disk-usage",
"metric": "disk-usage",
"threshold": 80,
"value": 82.9,
"previousValue": 82.8,
"trend": "rising",
"firstSeenAt": "2026-01-01T00:00:00.000Z",
"repeatCount": 4,
"untrusted": {
"affected": [],
"topContributors": [
{ "kind": "table", "database": "db_a", "table": "events", "bytesOnDisk": 120000000000 }
]
}
}
}firstSeenAtis the first alert-history event of this episode (since the rule last recovered on that host). Breaches still waiting on hysteresis write no history, so the condition may be up tominConsecutiveBreachessweeps older. It isnullwhen the deployment has no alert-history store.topContributorsis sent only to custom targets you configure (env,routing.yaml, alert routes). Webhook subscriptions are created by signed-in users and never receive it, because it carries other users' normalized SQL, ClickHouse user names and query ids. Subscriptions get the rest of the block, includingaffected.untrustedholds strings read from your cluster (table names, users, normalized queries). They are truncated to 200 characters and passed through the same secret redaction as agent errors. If you feed the block to an LLM, treatuntrustedas data, never as instructions.- Nothing runs on chmonitor's side to build it: no LLM call and no write. At most one read-only ClickHouse query (top contributors) and one alert-history read per alert, shared by every opted-in destination.
Slack app: one standing message, changes in its thread
A plain webhook can only post new messages. If the chmonitor Slack app is installed in the same workspace, a webhook alert route can name a Slack channel id, and alerts for that route post through the app instead. The route keeps one standing message that is edited in place:
- The message shows the open counts, for example
🔴 3 open · 1 critical · 2 warning, and is edited every sweep that changes them. - The thread gets what changed: one finding's details (host, metric, value
and thresholds, runbooks, ack button), a
+1 warning · 3 openline, or a digest when several new rules fire together. - All clear: when nothing is open, the message is edited to
✅ All clear. The next incident starts a fresh message. - If the stored message was deleted, a new one is posted and used from then on.
To turn it on:
- Install the Slack app (it requests the
chat:writescope) and invite it to the channel:/invite @chmonitor. - Copy the channel id: open the channel details in Slack; the id at the
bottom looks like
C0123456789. - Add a route with the channel's incoming-webhook URL and the channel id. In
the UI, the Slack channel id field shows once the URL is a
hooks.slack.comURL. Inrouting.yaml:
routes:
- id: slack-alerts
provider: webhook
secretEnv: SLACK_ALERTS_WEBHOOK_URL
target:
slackChannelId: C0123456789The app is matched by the workspace id in the webhook URL. The webhook stays the fallback: with no channel id, no installed app for that workspace, no metadata database (the edit needs the stored message id), or any Slack API error, the alert posts through the webhook exactly as before, and the error is logged. Alerts held by the digest time window also keep using the webhook.
3. Test it
curl -H "Authorization: Bearer $CRON_SECRET" \
https://your-chmonitor.example.com/api/cron/health-sweepThe endpoint always returns HTTP 200 with a JSON array of check results —
dispatch happens server-side and doesn't block the response. To confirm a
message actually went out, trip a threshold intentionally (temporarily lower
HEALTH_ALERT_MIN_SEVERITY to warning) and watch the channel.
Alert history
On Cloud with D1 configured, the Health page's alert history records every dispatch attempt and whether delivery succeeded. That history isn't available on a self-hosted deployment without D1.
Noise control: transitions, recovery & hysteresis
Alerts are transition-based: the sweep notifies only when a condition
actually changes state — ok → warning, warning → critical, or recovers back
to ok. A condition that stays unhealthy is not re-sent on every run; it sends a
single reminder at most once per HEALTH_ALERT_COOLDOWN_MINUTES (default 60;
0 disables reminders). A recovery message always goes out when a condition
clears, and reports how long the incident was firing.
Slack and generic webhook URLs get a digest only when the set of open alerts
changed. A sweep where nothing changed sends nothing, and these channels get no
reminders. A new rule or a higher severity sends the full digest. When alerts
only resolve, or an already-open rule fires on another host, the channel gets
one short line such as +1 warning · 1 resolved · 60 open. Other channels keep
their reminders.
Two per-rule knobs tune this without touching the rest of your alerting:
# 0 = alert once, then stay quiet until it escalates or recovers.
HEALTH_ALERT_COOLDOWN_DISK_USAGE_MINUTES=0
# Mute checks on every channel. They still run and show on the Health page.
HEALTH_ALERT_MUTED_RULES=replication-lag,merges-*Custom rules (custom:...) cannot be named in an env var, so they always use
the global cooldown; they can still be muted by their full id.
To stop a metric that hovers on a threshold from flapping (fire/recover on every sweep), each check applies hysteresis:
# Consecutive breaches required before an alert fires (default 1 — fire promptly).
HEALTH_HYSTERESIS_BREACHES=1
# Consecutive "ok" sweeps required before a firing condition recovers (default 2).
HEALTH_HYSTERESIS_CLEARS=2
# Per-check overrides use the rule id, uppercased with dashes → underscores:
HEALTH_HYSTERESIS_DISK_USAGE_CLEARS=3The default (fire on the first breach, recover only after two consecutive clears) keeps genuine criticals prompt while collapsing a flapping metric into exactly one fire and one recover. The current state and last transition per check are shown on the Active tab of Alert Settings. State is persisted to D1 (when configured) so hysteresis streaks and incident timers survive worker restarts.
Kubernetes (Helm)
On Kubernetes, the Helm chart runs the sweep for you. Turn on cron and the
chart creates a CronJob that calls /api/cron/health-sweep every 5 minutes. See
Deploy on Kubernetes for the rest of the chart.
Create the Secret first. The webhook URL is a credential, so it never goes in
values.yaml:
kubectl create secret generic chmonitor-alert-secrets \
--from-literal=CRON_SECRET="$(openssl rand -hex 32)" \
--from-literal=slack-webhook-url='https://hooks.slack.com/services/T000/B000/XXXX'Then set the values:
# values.yaml
cron:
enabled: true
existingSecret: chmonitor-alert-secrets # must contain the key CRON_SECRET
schedules:
healthSweep: "*/5 * * * *"
alertWebhooks:
enabled: true
defaults:
format: slack
minSeverity: warning
targets:
- name: team-slack
format: slack
urlFrom:
name: chmonitor-alert-secrets
key: slack-webhook-url
extraEnv:
- name: HEALTH_ALERT_MIN_SEVERITY
value: "warning" # or "critical"The chart has no dedicated value for HEALTH_ALERT_MIN_SEVERITY, so it goes
through extraEnv. Each target also has its own minSeverity.
The dashboard needs the same CRON_SECRET
The CronJob and the dashboard container read CRON_SECRET from the same
Secret. If they differ, the sweep is rejected.
Plain env alternative
If you do not want alertWebhooks, use the single global webhook. Keep the URL
in a Secret and read it with secretKeyRef:
extraEnv:
- name: HEALTH_ALERT_WEBHOOK_URL
valueFrom:
secretKeyRef:
name: chmonitor-alert-secrets
key: slack-webhook-url
- name: HEALTH_ALERT_MIN_SEVERITY
value: "warning"Disable or pause alerts
Pick the narrowest switch that does the job.
| Switch | Scope | Notes |
|---|---|---|
HEALTH_ALERT_ENABLED=false | All channels | Master switch. Keeps the URLs, so you can turn it back on. |
HEALTH_ALERT_WEBHOOK_ENABLED=false | The webhook channel only | Covers routes and the global HEALTH_ALERT_WEBHOOK_URL. Other channels use HEALTH_ALERT_<CHANNEL>_ENABLED with EMAIL, OPSGENIE, PAGERDUTY, TELEGRAM, NTFY or PUSHOVER. |
enabled: false on a target | One custom webhook target | In Helm values or in Health settings. A disabled UI row also hides the Helm target with the same name. |
alertWebhooks.enabled=false | All Helm-declared targets | The global HEALTH_ALERT_WEBHOOK_URL keeps working. |
cron.enabled=false | Scheduled sweeps | No CronJob, so nothing runs on a schedule. Manual curl calls still work. |
CHM_HEALTH_SWEEP_ENABLED=false | Scheduled sweeps (headless) | Pauses the scheduled sweep without removing the cron. |
| Quiet hours | Recurring time windows | Health settings, Advanced tab. See Health. |
| Maintenance windows | One host or all hosts, for a set time | Health settings, Advanced tab. See Health. |
HEALTH_ALERT_MIN_SEVERITY=critical | All channels | Does not pause anything. It only drops warnings, which cuts noise. |
Several deployments in one channel
If many chmonitor deployments post to the same channel, each message must say where it came from.
- Set
CLICKHOUSE_NAMEfor each host. The host label in the message uses it, soprod-eu-1reads better than0. - Use one webhook target per deployment, with its own
nameand channel.
You can also tag every message from a deployment with these variables:
| Variable | Effect |
|---|---|
HEALTH_ALERT_INSTANCE_NAME | Adds a [prod-eu] prefix to the title. With Hookshot, it becomes the username. |
HEALTH_ALERT_DASHBOARD_URL | Adds an "Open dashboard" link to /health?host=N. |
HEALTH_ALERT_METADATA | key=value pairs, for example team=data,region=eu. At most 10 pairs. They show in the Slack context line and as JSON fields. |
In a title or body template you can use {{instance}}, {{dashboardUrl}} and
{{meta.<key>}}, for example {{meta.team}}.
In Helm, set the same values under alerting:
alerting:
instanceName: prod-eu
dashboardUrl: https://chmonitor.example.com
metadata:
team: data
region: euMessage format examples
The exact text depends on the check. These show the shape.
Slack (sent as blocks, shown here as the visible text):
[CRITICAL] Disk usage high
Disk usage on default is 93%
host prod-eu-1 · disk_usage = 93 · 2026-10-11T08:00:00ZHookshot / Element:
{
"text": "🔴 [CRITICAL] Disk usage high — Disk usage on default is 93%\nhost prod-eu-1 · disk-usage-percent = 93 · 2026-10-11T08:00:00Z",
"html": "<b><font data-mx-color=\"#dd3333\">🔴 [CRITICAL] Disk usage high</font></b><br>Disk usage on default is 93%<br><i>host prod-eu-1 · disk-usage-percent = 93 · 2026-10-11T08:00:00Z</i>"
}Raw JSON (a normalized payload for a generic receiver):
{
"severity": "critical",
"title": "Disk usage high",
"metric": "disk-usage-percent",
"value": 93,
"thresholds": { "warning": 80, "critical": 90 },
"host": { "id": 0, "label": "prod-eu-1" },
"label": "Disk usage on default is 93%",
"runbookUrls": [],
"timestamp": "2026-10-11T08:00:00Z",
"text": "[CRITICAL] Disk usage high — Disk usage on default is 93% (host prod-eu-1)"
}With HEALTH_ALERT_INSTANCE_NAME, HEALTH_ALERT_DASHBOARD_URL or
HEALTH_ALERT_METADATA set, the body also carries instance, dashboardUrl
and metadata.
Custom title and body
A target can set titleTemplate and textTemplate. Available variables:
{{severity}}, {{title}}, {{label}}, {{host}}, {{hostId}},
{{checkId}}, {{value}} and {{timestamp}}, plus {{instance}},
{{dashboardUrl}} and {{meta.<key>}}. These facts are empty when they do
not apply:
| Variable | Value |
|---|---|
{{warn}}, {{crit}} | The warning and critical thresholds. |
{{thresholds}} | Both thresholds, e.g. warn 80 | crit 90. |
{{runbook}} | The first runbook URL. |
{{status}} | Reminder: still open, 3 h (82.8 -> 82.9). Recovery: fired for 3 h. |
{{stillOpen}} | Reminder only: how long the incident has been open, e.g. 3 h. |
{{previousValue}} | Reminder only: the value at the previous notification. |
{{firedFor}} | Recovery only: how long the incident fired, e.g. 3 h. |
{{firstFired}} | When the condition was first seen (alert.firstSeenAt), as an ISO time. Needs details: true on the target. |
An unknown name is left as written, so a typo is easy to spot in Preview.
alertWebhooks:
targets:
- name: team-slack
format: slack
urlFrom:
name: chmonitor-alert-secrets
key: slack-webhook-url
titleTemplate: "[{{instance}}] {{severity}}: {{title}}"
textTemplate: "{{label}} on {{host}} ({{checkId}} = {{value}}) at {{timestamp}} - {{dashboardUrl}} (team {{meta.team}})"Related
- Health alerts on Kubernetes — a step-by-step Helm setup with a webhook, tests and troubleshooting.
- Health — every check, threshold, and environment variable, plus scheduling on Cloudflare Cron.
- Slack app — the native
/chmonitorSlack app (slash command, Home tab, Acknowledge buttons), a different and more capable integration than a plain incoming webhook.
Slack app
Install the native chmonitor Slack app — OAuth setup, the /chmonitor slash command (status, read-only query, alerts), a Home tab, and Acknowledge buttons on alert messages.
Health alerts on Kubernetes
Step by step, set up health alerts with a webhook on Kubernetes using the Helm chart, then test them, share one channel across deployments, and fix common problems.