rules & escalation
A monitor that fails silently isn't worth much. A rule connects a
monitor's state changes to the
channels that should hear about them. Rules
live at /rules.
fields
| field | meaning |
|---|---|
| name | label for the rule |
| monitors | which monitors it applies to — every monitor in the org, or a specific multi-selected set |
| triggers | a multi-select subset of on_warning, on_failure, on_recovery (at least one) |
| channels | the channels notified immediately (the first tier) |
| failure threshold | notify only after N consecutive checks at the triggering severity. Default 1 = fire on the first |
| re-notify minutes | resend every N minutes while the incident stays open at or above the triggering severity. 0 disables |
| escalation | up to 3 further tiers, each with a delay and its own channels |
triggers encode severity
There's no separate severity filter — the trigger is the severity:
- on_warning fires on any transition where the new severity is
warning:
up → degraded, andcritical → warningwithin an incident. - on_failure fires on any transition where the new severity is
critical:
up → down, andwarning → critical. - on_recovery fires on
* → up— the incident closing.
Triggers are multi-select because the common case is one rule that should fire on every change in a monitor's lifecycle and route to one channel.
One rule, whole lifecycle: triggers: [on_warning, on_failure, on_recovery] → one Slack channel covers warn, fail, and recovery.
Split routing: two rules — [on_warning] → Slack and [on_failure]
→ PagerDuty — send warnings to chat and pages to your pager.
escalation
Beyond the immediate channels (tier 1), a rule can add up to 3 escalation tiers. Each tier has a delay in minutes and its own set of channels: if the incident is still at or above the triggering severity when the delay elapses, those channels fire.
A typical pattern: tier 1 pings Slack on the first failure; tier 2, ten
minutes later, pages on-call if nobody's resolved it yet. Escalation
resets on recovery (* → up) — not on a critical → warning
de-escalation within the same incident.
deduplication & re-notify
Rules don't spam. The guarantee is one notification per
(incident, rule, trigger) unless escalation fires:
- A rule with
triggers: [on_warning, on_failure]sends at most one warning notification and at most one failure notification for a given incident — even if it bounces between degraded and down repeatedly. - An
up → degraded → down → upincident, with one rule whose triggers are all three, sends exactly three notifications: one warning, one failure, one recovery.
re-notify is the deliberate exception: set re-notify minutes to be reminded every N minutes while the incident stays open at or above the rule's triggering severity.
For a hands-on walkthrough of building these, see alerting & escalation.