alerting & escalation

This guide builds a rule from scratch. You'll need at least one channel already set up.

1. pick what it covers

/rules → + new rule. First choose the scope:

• every monitor in this org — the rule follows your whole fleet, including monitors you add later. Good for a baseline "page us if anything goes down" rule.
• specific monitors — multi-select the ones this rule applies to. Good for routing critical services differently from the rest.

2. choose triggers

Select one or more of on_warning, on_failure, on_recovery. Remember the trigger encodes the severity:

• on_failure fires when a monitor goes (or escalates to) down.
• on_warning fires when it goes (or de-escalates to) degraded.
• on_recovery fires when it returns to up.

3. set the failure threshold

failure threshold is how many consecutive checks at the triggering severity must occur before the rule fires. Default 1 = fire on the first. Bump it to 2–3 for a flaky endpoint so a single blip doesn't page anyone.

4. pick the immediate channels

Choose the channel(s) notified the moment the rule fires. This is tier 1.

5. add escalation (optional)

Add up to 3 escalation tiers. Each tier has a delay in minutes and its own channels. If the incident is still at or above the triggering severity when a tier's delay elapses, that tier's channels fire.

A common two-tier shape:

tier 1 (0 min)   →  ops-slack          fire immediately
tier 2 (+10 min) →  oncall-pagerduty    page if still unresolved

Escalation resets on recovery — once the monitor returns to up, the escalation clock is cleared. It does not reset on a critical → warning de-escalation within the same incident.

6. tune re-notify

re-notify minutes resends the alert every N minutes while the incident stays open at or above the triggering severity. 0 disables it. Use a small value (5–15) for pager-style nagging on critical services; leave it off for channels where one ping is enough.

what you can rely on

• No duplicate spam. One notification per (incident, rule, trigger) unless escalation or re-notify fires. A monitor flapping between degraded and down won't re-page on every flip.
• One incident, full story. A degraded → down → recovered sequence is a single incident with severity transitions recorded — your rule fires per trigger, not per flip.

next

• Rules & escalation concepts — the dedup and severity model in full.
• Manage it as code — declare monitors, channels, and rules in one YAML file.