intermediate 2 min answer

An on-call rotation is producing burnout and slow responses. Alert volume is high and most pages are not actionable. What changes, and in what order?

sliceoncallalertingactionabilitysustainability
Show the full answer Hide the answer

The order of changes

1. Delete alerts that are not actionable. The single highest-impact change and the one teams resist, because every alert was added by someone who thought it mattered. The test is concrete: if this fires at 3am, is there something a human must do right now? If not, it is a dashboard entry or a ticket, not a page.

2. Alert on symptoms, not causes. One page saying "checkout success rate below threshold" is worth twenty saying a specific host has high CPU. Cause-based alerts fire in clusters during a single incident and bury the signal.

3. Route by severity. Pages wake people; tickets do not. A large fraction of what currently pages is genuinely a ticket, and reclassifying it costs nothing.

4. Fix the top recurring page. Alert volume is usually dominated by a small number of sources, and the fastest way to reduce load is to remove the cause of the most frequent one rather than to tune thresholds.

5. Then adjust the rotation — more people, shorter shifts, follow-the-sun. Doing this first spreads the pain without reducing it, which is why it is last.

The measurements to run

  • Pages per shift, with a target the team agrees is sustainable. A number nobody has set will not be met.
  • Percentage of pages that resulted in action, which measures actionability directly.
  • Time to acknowledge and time to mitigate, which distinguishes an alerting problem from a diagnosis problem.
  • Pages by source, which identifies the top offender.

The structural conditions

  • The team that builds owns the pager, or alert quality has no feedback loop — an operations team paged by someone else's code cannot fix the cause.
  • Time to fix the alerts allocated explicitly, because it always loses to feature work when negotiated per item.
  • Compensation or time off in lieu, which makes the cost of a noisy rotation visible to the people who can authorise fixing it.

The framing that matters

A noisy pager is an architectural signal. Persistent alerting on the same class of failure means the system requires human intervention to stay up, and the correct response is to remove the need rather than to schedule the intervention more sustainably. On-call load is a proxy for how much the design depends on people.