Monitoring that pages people only when something is actually wrong

The economics of a page

Every alert that wakes an engineer and requires no action makes the next real alert slightly less believed. Teams do not decide to ignore their monitoring; they are trained to, one false page at a time. Designing alerting is therefore mostly about deciding what not to alert on.

The reliable rule: page on symptoms, not causes. Users experience error rates, latency, and unavailability — so those are the conditions worth interrupting a human for. High CPU, a full-ish disk, a restarted container are causes; they belong on dashboards and in tickets unless they are actively producing symptoms.

Burn rates instead of thresholds

Static thresholds produce both false alarms and late alarms because they ignore duration. Error-budget burn rates fix this: define the reliability target the service actually promises, then alert when the current failure rate would exhaust that budget too quickly. A fast, severe burn pages immediately; a slow burn opens a ticket for working hours.

This also forces the conversation that matters — what does this service actually promise? A back-office report generator and a checkout flow should not share an alerting posture, and burn-rate alerting makes that difference explicit instead of tribal.

Every page ends in a runbook

An alert without a runbook outsources system knowledge to whoever happens to be on call. Each paging alert should link to a short document: what the alert means, how to confirm impact, the first three diagnostic steps, and the escalation path. Writing the runbook is also the cheapest review of the alert itself — if there is no clear action to write, the alert should not page anyone.

• Page on user-facing symptoms; dashboard the causes
• Alert on error-budget burn rate, not raw thresholds
• No paging alert ships without a runbook and an owner