chmonitorchmonitor
Guides

Guide

Alerting to Slack and Discord

Wire chmonitor's health sweep to a Slack or Discord incoming webhook so your team gets notified the moment a check crosses a threshold — no polling the dashboard required.

9 min read

chmonitor's Health page checks replication lag, failed mutations, disk usage and a handful of other signals on a schedule you control. Left alone, that only helps the person who remembers to look at the page. The health-sweep cron endpoint closes that gap: point it at a Slack or Discord incoming webhook and it posts a message whenever a check meets or exceeds a severity threshold.

This page walks through creating the webhook URL itself. For the full CRON_SECRET / HEALTH_ALERT_* environment variable reference, scheduling on Cloudflare Cron, and the sequence diagram, see Health → Health-sweep cron alerting.

Prerequisites

  • A chmonitor instance already connected to at least one ClickHouse host.
  • The ability to set environment variables / secrets on your deployment (Cloudflare Worker secret, Docker env, or Kubernetes Secret).

1. Create the webhook URL

  1. Go to api.slack.com/apps → create (or open) an app → Incoming Webhooks → turn the toggle on.
  2. Add New Webhook to Workspace, pick the channel, and copy the generated URL (https://hooks.slack.com/services/...).
  1. In the target channel: Channel Settings → Integrations → Webhooks → New Webhook.
  2. Name it, optionally set an avatar, and Copy Webhook URL.

The same URL works for both — the sweep posts {"text": "...", "content": "..."}, and each platform reads the field it understands.

2. Configure and enable

CRON_SECRET=<random-secret>
HEALTH_ALERT_MIN_SEVERITY=warning   # or "critical" to only page on the worst checks
HEALTH_ALERT_WEBHOOK_URL=<the URL you copied above>

On Cloudflare Workers, set the secrets rather than plain vars:

wrangler secret put CRON_SECRET
wrangler secret put HEALTH_ALERT_WEBHOOK_URL

HEALTH_ALERT_ENABLED is optional. Alerting defaults to on as soon as HEALTH_ALERT_WEBHOOK_URL is set; set HEALTH_ALERT_ENABLED=false to pause alerts without removing the URL. The same rule applies to email (HEALTH_ALERT_EMAIL_ENABLED): it is on once the provider URL, from and to addresses are set.

Custom formats and multiple targets

For more control, open Health settings → Alerts → Custom webhook targets. Each target can select:

  • Raw JSON — the normalized alert payload for a generic receiver.
  • Slack — escaped Slack blocks.
  • Element / Matrix — an m.notice event with escaped HTML.
  • Auto — preserve the legacy URL-based format detection.
  • Matrix Hookshot — the {text, html} body that matrix-hookshot generic webhooks expect. Use this, not Element / Matrix, for a Hookshot URL: Hookshot posts any body without a text key to the room as raw JSON.

Matrix Hookshot and Element / Matrix targets with no title or body template are grouped per sweep: when one sweep finds several alerts for the same target, the room gets ONE message. It starts with a coloured summary line (3 critical, 1 warning on 4 hosts), lists the findings under each rule (host, metric = value, severity), stops after 25 lines with …and N more, and ends with one dashboard link. A single finding is sent exactly as before. Grouped targets are sent in the same sweep even when the time-window digest is on; they are never written to the digest buffer, because they can carry secret headers. A target with a template sends one message per finding.

A Slack format target sends one compact line per alert next to a severity colour bar, for example 🔴 CRITICAL Stuck Merges · node-2 · 4 (warn 1, crit 3) · Open. A recovery reads 🟢 RESOLVED … · fired for 35 min · Open. The instance prefix appears only when instanceName is set; runbook links and metadata go on one small second line when there are any. A grouped Slack digest is compact text: a bold summary line, one line per rule in the same layout as a single alert, and a small gray footer. Several alerts of one rule and severity share one line: the rule title with a count, up to three names then +N, and the age of the oldest alert, for example 🟠 WARNING Source stopped sending rows (34) · db.events:source-1, db.events:source-2, db.events:source-3, +31 · 3 h. A lone alert shows its full timeline (still open, 3 h (88.5 -> 97.25), fired for 35 min), or new. Recoveries get their own line. The summary line counts every alert. When the lines share one host, the summary names it and the lines leave it out; otherwise each line adds the hosts. Past 25 lines the digest ends with …and N more. The footer holds the Open dashboard link, up to three runbook links, the instance name and metadata, and the sweep time in UTC. A sweep that only adds alerts of a rule that is already open, or only resolves, posts one line instead of the digest: +1 warning · 1 resolved · 60 open.

The form supports bounded {{severity}}, {{title}}, {{label}}, {{host}}, {{value}}, and {{timestamp}} templates plus safe X-* headers. Preview uses the same server-side formatter as delivery and masks the destination URL; Send test is an explicit write-gated request. Target URLs are credentials: they are never returned to the browser, and authorization headers belong in a Kubernetes Secret or another server-side secret source.

On Kubernetes, the Helm chart can declare the same targets under alertWebhooks.targets for GitOps-managed deployments. Use urlFrom and headersSecretFrom with an existing Secret; do not commit token-bearing URLs in values.yaml, a ConfigMap, or extraEnv. D1/UI configuration is owner-scoped and takes precedence over a Helm target with the same name; deleting the D1 row returns to the Helm declaration.

Structured alert block for agents

Set details: true on a Raw JSON custom target, or on a webhook subscription ("details": true in POST/PATCH /api/v1/webhooks/subscriptions), to add one alert block to the body. Chat receivers and agents read the same schema. Off (the default), the body is unchanged. Subscriptions carry it as data.alert on alert.fired and alert.resolved.

{
  "alert": {
    "ruleId": "disk-usage",
    "metric": "disk-usage",
    "threshold": 80,
    "value": 82.9,
    "previousValue": 82.8,
    "trend": "rising",
    "firstSeenAt": "2026-01-01T00:00:00.000Z",
    "repeatCount": 4,
    "untrusted": {
      "affected": [],
      "topContributors": [
        { "kind": "table", "database": "db_a", "table": "events", "bytesOnDisk": 120000000000 }
      ]
    }
  }
}
  • firstSeenAt is the first alert-history event of this episode (since the rule last recovered on that host). Breaches still waiting on hysteresis write no history, so the condition may be up to minConsecutiveBreaches sweeps older. It is null when the deployment has no alert-history store.
  • topContributors is sent only to custom targets you configure (env, routing.yaml, alert routes). Webhook subscriptions are created by signed-in users and never receive it, because it carries other users' normalized SQL, ClickHouse user names and query ids. Subscriptions get the rest of the block, including affected.
  • untrusted holds strings read from your cluster (table names, users, normalized queries). They are truncated to 200 characters and passed through the same secret redaction as agent errors. If you feed the block to an LLM, treat untrusted as data, never as instructions.
  • Nothing runs on chmonitor's side to build it: no LLM call and no write. At most one read-only ClickHouse query (top contributors) and one alert-history read per alert, shared by every opted-in destination.

Slack app: one standing message, changes in its thread

A plain webhook can only post new messages. If the chmonitor Slack app is installed in the same workspace, a webhook alert route can name a Slack channel id, and alerts for that route post through the app instead. The route keeps one standing message that is edited in place:

  • The message shows the open counts, for example 🔴 3 open · 1 critical · 2 warning, and is edited every sweep that changes them.
  • The thread gets what changed: one finding's details (host, metric, value and thresholds, runbooks, ack button), a +1 warning · 3 open line, or a digest when several new rules fire together.
  • All clear: when nothing is open, the message is edited to ✅ All clear. The next incident starts a fresh message.
  • If the stored message was deleted, a new one is posted and used from then on.

To turn it on:

  1. Install the Slack app (it requests the chat:write scope) and invite it to the channel: /invite @chmonitor.
  2. Copy the channel id: open the channel details in Slack; the id at the bottom looks like C0123456789.
  3. Add a route with the channel's incoming-webhook URL and the channel id. In the UI, the Slack channel id field shows once the URL is a hooks.slack.com URL. In routing.yaml:
routes:
  - id: slack-alerts
    provider: webhook
    secretEnv: SLACK_ALERTS_WEBHOOK_URL
    target:
      slackChannelId: C0123456789

The app is matched by the workspace id in the webhook URL. The webhook stays the fallback: with no channel id, no installed app for that workspace, no metadata database (the edit needs the stored message id), or any Slack API error, the alert posts through the webhook exactly as before, and the error is logged. Alerts held by the digest time window also keep using the webhook.

3. Test it

curl -H "Authorization: Bearer $CRON_SECRET" \
  https://your-chmonitor.example.com/api/cron/health-sweep

The endpoint always returns HTTP 200 with a JSON array of check results — dispatch happens server-side and doesn't block the response. To confirm a message actually went out, trip a threshold intentionally (temporarily lower HEALTH_ALERT_MIN_SEVERITY to warning) and watch the channel.

Alert history

On Cloud with D1 configured, the Health page's alert history records every dispatch attempt and whether delivery succeeded. That history isn't available on a self-hosted deployment without D1.

Noise control: transitions, recovery & hysteresis

Alerts are transition-based: the sweep notifies only when a condition actually changes state — ok → warning, warning → critical, or recovers back to ok. A condition that stays unhealthy is not re-sent on every run; it sends a single reminder at most once per HEALTH_ALERT_COOLDOWN_MINUTES (default 60; 0 disables reminders). A recovery message always goes out when a condition clears, and reports how long the incident was firing.

Slack and generic webhook URLs get a digest only when the set of open alerts changed. A sweep where nothing changed sends nothing, and these channels get no reminders. A new rule or a higher severity sends the full digest. When alerts only resolve, or an already-open rule fires on another host, the channel gets one short line such as +1 warning · 1 resolved · 60 open. Other channels keep their reminders.

Two per-rule knobs tune this without touching the rest of your alerting:

# 0 = alert once, then stay quiet until it escalates or recovers.
HEALTH_ALERT_COOLDOWN_DISK_USAGE_MINUTES=0

# Mute checks on every channel. They still run and show on the Health page.
HEALTH_ALERT_MUTED_RULES=replication-lag,merges-*

Custom rules (custom:...) cannot be named in an env var, so they always use the global cooldown; they can still be muted by their full id.

To stop a metric that hovers on a threshold from flapping (fire/recover on every sweep), each check applies hysteresis:

# Consecutive breaches required before an alert fires (default 1 — fire promptly).
HEALTH_HYSTERESIS_BREACHES=1
# Consecutive "ok" sweeps required before a firing condition recovers (default 2).
HEALTH_HYSTERESIS_CLEARS=2

# Per-check overrides use the rule id, uppercased with dashes → underscores:
HEALTH_HYSTERESIS_DISK_USAGE_CLEARS=3

The default (fire on the first breach, recover only after two consecutive clears) keeps genuine criticals prompt while collapsing a flapping metric into exactly one fire and one recover. The current state and last transition per check are shown on the Active tab of Alert Settings. State is persisted to D1 (when configured) so hysteresis streaks and incident timers survive worker restarts.

Kubernetes (Helm)

On Kubernetes, the Helm chart runs the sweep for you. Turn on cron and the chart creates a CronJob that calls /api/cron/health-sweep every 5 minutes. See Deploy on Kubernetes for the rest of the chart.

Create the Secret first. The webhook URL is a credential, so it never goes in values.yaml:

kubectl create secret generic chmonitor-alert-secrets \
  --from-literal=CRON_SECRET="$(openssl rand -hex 32)" \
  --from-literal=slack-webhook-url='https://hooks.slack.com/services/T000/B000/XXXX'

Then set the values:

# values.yaml
cron:
  enabled: true
  existingSecret: chmonitor-alert-secrets   # must contain the key CRON_SECRET
  schedules:
    healthSweep: "*/5 * * * *"

alertWebhooks:
  enabled: true
  defaults:
    format: slack
    minSeverity: warning
  targets:
    - name: team-slack
      format: slack
      urlFrom:
        name: chmonitor-alert-secrets
        key: slack-webhook-url

extraEnv:
  - name: HEALTH_ALERT_MIN_SEVERITY
    value: "warning"          # or "critical"

The chart has no dedicated value for HEALTH_ALERT_MIN_SEVERITY, so it goes through extraEnv. Each target also has its own minSeverity.

The dashboard needs the same CRON_SECRET

The CronJob and the dashboard container read CRON_SECRET from the same Secret. If they differ, the sweep is rejected.

Plain env alternative

If you do not want alertWebhooks, use the single global webhook. Keep the URL in a Secret and read it with secretKeyRef:

extraEnv:
  - name: HEALTH_ALERT_WEBHOOK_URL
    valueFrom:
      secretKeyRef:
        name: chmonitor-alert-secrets
        key: slack-webhook-url
  - name: HEALTH_ALERT_MIN_SEVERITY
    value: "warning"

Disable or pause alerts

Pick the narrowest switch that does the job.

SwitchScopeNotes
HEALTH_ALERT_ENABLED=falseAll channelsMaster switch. Keeps the URLs, so you can turn it back on.
HEALTH_ALERT_WEBHOOK_ENABLED=falseThe webhook channel onlyCovers routes and the global HEALTH_ALERT_WEBHOOK_URL. Other channels use HEALTH_ALERT_<CHANNEL>_ENABLED with EMAIL, OPSGENIE, PAGERDUTY, TELEGRAM, NTFY or PUSHOVER.
enabled: false on a targetOne custom webhook targetIn Helm values or in Health settings. A disabled UI row also hides the Helm target with the same name.
alertWebhooks.enabled=falseAll Helm-declared targetsThe global HEALTH_ALERT_WEBHOOK_URL keeps working.
cron.enabled=falseScheduled sweepsNo CronJob, so nothing runs on a schedule. Manual curl calls still work.
CHM_HEALTH_SWEEP_ENABLED=falseScheduled sweeps (headless)Pauses the scheduled sweep without removing the cron.
Quiet hoursRecurring time windowsHealth settings, Advanced tab. See Health.
Maintenance windowsOne host or all hosts, for a set timeHealth settings, Advanced tab. See Health.
HEALTH_ALERT_MIN_SEVERITY=criticalAll channelsDoes not pause anything. It only drops warnings, which cuts noise.

Several deployments in one channel

If many chmonitor deployments post to the same channel, each message must say where it came from.

  • Set CLICKHOUSE_NAME for each host. The host label in the message uses it, so prod-eu-1 reads better than 0.
  • Use one webhook target per deployment, with its own name and channel.

You can also tag every message from a deployment with these variables:

VariableEffect
HEALTH_ALERT_INSTANCE_NAMEAdds a [prod-eu] prefix to the title. With Hookshot, it becomes the username.
HEALTH_ALERT_DASHBOARD_URLAdds an "Open dashboard" link to /health?host=N.
HEALTH_ALERT_METADATAkey=value pairs, for example team=data,region=eu. At most 10 pairs. They show in the Slack context line and as JSON fields.

In a title or body template you can use {{instance}}, {{dashboardUrl}} and {{meta.<key>}}, for example {{meta.team}}.

In Helm, set the same values under alerting:

alerting:
  instanceName: prod-eu
  dashboardUrl: https://chmonitor.example.com
  metadata:
    team: data
    region: eu

Message format examples

The exact text depends on the check. These show the shape.

Slack (sent as blocks, shown here as the visible text):

[CRITICAL] Disk usage high
Disk usage on default is 93%
host prod-eu-1 · disk_usage = 93 · 2026-10-11T08:00:00Z

Hookshot / Element:

{
  "text": "🔴 [CRITICAL] Disk usage high — Disk usage on default is 93%\nhost prod-eu-1 · disk-usage-percent = 93 · 2026-10-11T08:00:00Z",
  "html": "<b><font data-mx-color=\"#dd3333\">🔴 [CRITICAL] Disk usage high</font></b><br>Disk usage on default is 93%<br><i>host prod-eu-1 · disk-usage-percent = 93 · 2026-10-11T08:00:00Z</i>"
}

Raw JSON (a normalized payload for a generic receiver):

{
  "severity": "critical",
  "title": "Disk usage high",
  "metric": "disk-usage-percent",
  "value": 93,
  "thresholds": { "warning": 80, "critical": 90 },
  "host": { "id": 0, "label": "prod-eu-1" },
  "label": "Disk usage on default is 93%",
  "runbookUrls": [],
  "timestamp": "2026-10-11T08:00:00Z",
  "text": "[CRITICAL] Disk usage high — Disk usage on default is 93% (host prod-eu-1)"
}

With HEALTH_ALERT_INSTANCE_NAME, HEALTH_ALERT_DASHBOARD_URL or HEALTH_ALERT_METADATA set, the body also carries instance, dashboardUrl and metadata.

Custom title and body

A target can set titleTemplate and textTemplate. Available variables: {{severity}}, {{title}}, {{label}}, {{host}}, {{hostId}}, {{checkId}}, {{value}} and {{timestamp}}, plus {{instance}}, {{dashboardUrl}} and {{meta.<key>}}. These facts are empty when they do not apply:

VariableValue
{{warn}}, {{crit}}The warning and critical thresholds.
{{thresholds}}Both thresholds, e.g. warn 80 | crit 90.
{{runbook}}The first runbook URL.
{{status}}Reminder: still open, 3 h (82.8 -> 82.9). Recovery: fired for 3 h.
{{stillOpen}}Reminder only: how long the incident has been open, e.g. 3 h.
{{previousValue}}Reminder only: the value at the previous notification.
{{firedFor}}Recovery only: how long the incident fired, e.g. 3 h.
{{firstFired}}When the condition was first seen (alert.firstSeenAt), as an ISO time. Needs details: true on the target.

An unknown name is left as written, so a typo is easy to spot in Preview.

alertWebhooks:
  targets:
    - name: team-slack
      format: slack
      urlFrom:
        name: chmonitor-alert-secrets
        key: slack-webhook-url
      titleTemplate: "[{{instance}}] {{severity}}: {{title}}"
      textTemplate: "{{label}} on {{host}} ({{checkId}} = {{value}}) at {{timestamp}} - {{dashboardUrl}} (team {{meta.team}})"
  • Health alerts on Kubernetes — a step-by-step Helm setup with a webhook, tests and troubleshooting.
  • Health — every check, threshold, and environment variable, plus scheduling on Cloudflare Cron.
  • Slack app — the native /chmonitor Slack app (slash command, Home tab, Acknowledge buttons), a different and more capable integration than a plain incoming webhook.

On this page