advanced 2 min answer

Your team receives 200 pages a week and the on-call rotation has lost two engineers in six months. Fix it.

alertingon-callburn-rateculture
Show the full answer Hide the answer

Diagnose before deleting

Measure the actionability ratio: the proportion of pages that resulted in a human doing something. Recording an outcome at resolution — action taken, no action needed, duplicate, false positive — turns alert quality from an opinion into data and makes the deletion arguments easy to win.

Below 50% means the system is training people to ignore it. That is not carelessness; it is rational adaptation, and it cannot be fixed by asking people to try harder.

Immediate relief

Delete every alert with no action in the last 90 days. Not tune — delete. They can be recreated if something is missed, and almost none are.

Route the non-urgent to tickets. Disk full in six days, a certificate expiring in three weeks, an instance already replaced by autoscaling. All real, none warrant waking anyone.

Group related alerts so one cause produces one page rather than forty.

These three usually remove the majority of volume within a week, which buys the credibility for the structural work.

Rebuild on symptoms

Replace cause-based rules — high CPU, low disk, an unhealthy instance — with symptom-based ones: error rate, latency against the SLO, throughput collapse. Symptoms mean users are affected by definition, and they catch causes nobody wrote a rule for, which is most of them.

Causes stay useful on diagnostic dashboards linked from the alert.

Make severity proportionate

Multi-window burn-rate alerting. 14.4× over 1 hour with a 5-minute confirmation pages; 6× over 6 hours pages; 1× over 3 days becomes a ticket. The short window is what prevents paging about a problem that already resolved.

This ends the false choice between sensitive-and-noisy and quiet-and-blind.

Make the remaining alerts good

Every page must be actionable, carry a runbook link in the payload, name user impact rather than a component condition, and specify who it is routed to.

Fix the conditions that produced 200 pages

The volume is a symptom of unreliability, and deleting alerts does not fix that. Look for the top recurring causes and eliminate them — a flapping dependency, a service that needs a restart weekly, a batch job that fails predictably.

Track toil, cap it, and protect engineering time to reduce it. A team at 30% toil will be higher next year until nothing is left for the work that would reduce it.

Make it stick

A non-actionable page is a defect — filed and fixed like any bug.

Review alert routing after every incident, in both directions. What paged and needed nothing becomes a ticket; what was a ticket and should have woken someone becomes a page.

Report the actionability ratio and pages per shift alongside other team metrics, so the improvement is visible and the regression is caught.

What a strong answer adds

Naming the human cost directly. Two engineers lost is the most expensive consequence in this scenario, and their system knowledge left with them. On-call sustainability is a retention issue, not an operational preference — and framing it that way is usually what gets the work funded.