Half your pages result in no action being taken. How do you fix that without reducing coverage?
Show the full answer Hide the answer
What is being tested
Whether you recognise that deleting non-actionable alerts increases effective coverage, and whether you know the alerting model that produces few, meaningful pages.
Reframing the premise
Those alerts are not coverage. An alert nobody acts on provides no protection, and it actively harms reliability by training responders to treat alerts as unimportant — so the one that mattered is missed. A monitoring system past that threshold has made the service less reliable.
Deleting them is not a trade-off against coverage. It is the removal of things that were never coverage.
The classification
Audit every alert that fired in the last month:
| Outcome | Action |
|---|---|
| No action taken | Delete |
| "Waited and it recovered" | Delete, or automate the remediation |
| Action needed but could have waited until morning | Downgrade to a ticket |
| Urgent human action required | Keep — this is a page |
Applied honestly, most alerts in a typical estate fall in the first three rows.
The model that produces good alerts
Alert on symptoms, not causes.
Cause-based alerting — CPU high, disk filling, queue growing — produces many alerts, most of which do not affect users. A CPU spike that nobody notices is not an incident.
Symptom-based alerting — the SLO is burning, checkout error rate is elevated, requests are timing out — produces few alerts, all of which matter. Causes belong on dashboards and in tickets; they are what you look at after a symptom alert fires.
The strongest form is error budget burn-rate alerting: page when the budget is being consumed fast enough to exhaust before the window ends. A fast burn pages immediately; a slow burn becomes a ticket. This calibrates urgency to actual user impact automatically, which static thresholds never do.
The supporting practices
- Group related alerts. One incident should page once, not thirty times. Alert on the causal symptom, not on every downstream effect.
- Every page has a runbook, or it requires original thought at 3am.
- Alert on age, not count. Queue depth is meaningless without throughput; the age of the oldest unprocessed item is directly meaningful.
- Track alert volume per shift as a metric and treat exceeding the target as a defect with an owner.
- Delete rather than tune. An alert kept "because it might be useful one day" costs attention every week and provides nothing.
What a strong answer adds
Naming the risk honestly: deleting alerts will eventually mean something is missed that an alert would have caught. That is a real cost, and it is smaller than the cost of a team that ignores alerts. Make the trade explicitly and monitor whether the SLO catches what the deleted alerts used to.