What is the test for whether an alert should exist, and what does applying it honestly do?
Show the full answer Hide the answer
What is being tested
Whether you can defend deleting most alerts as an increase in coverage rather than a reduction.
The test
Does a human need to take action, now, that cannot be automated?
| Outcome | Action |
|---|---|
| No action needed | Delete |
| Action needed but not urgent | Ticket, not a page |
| Action needed and automatable | Automate; alert only if the automation fails |
| Urgent human action required | Keep — this is a page |
What applying it honestly does
It deletes most alerts in a typical estate.
That is not a loss of coverage. An alert nobody acts on provides no protection and actively harms reliability, because it trains responders to treat alerts as unimportant — so the one that mattered is missed. A monitoring system past that threshold has made the service less reliable.
The model that produces good alerts
Alert on symptoms, not causes.
Cause-based alerting — CPU high, disk filling, queue growing — produces many alerts, most of which do not affect users. Symptom-based alerting — the SLO is burning, checkout errors are elevated, requests are timing out — produces few, all of which matter. Causes belong on dashboards; they are what you look at after a symptom fires.
The strongest form is error budget burn-rate alerting: page when the budget is being consumed fast enough to exhaust before the window ends; a slow burn becomes a ticket. That calibrates urgency to actual user impact automatically, which no static threshold does.
The supporting practices
- Group related alerts. One incident should page once, not thirty times.
- Every page has a runbook, or it requires original thought at 3am.
- Alert on age, not count — queue depth is meaningless without throughput.
- Track alert volume per shift and treat exceeding the target as a defect with an owner.
- Delete rather than tune. An alert kept "because it might be useful one day" costs attention every week.
The honest cost
Deleting alerts will eventually mean something is missed that an alert would have caught. That is a real cost, and it is smaller than the cost of a team that ignores alerts. Make the trade explicitly and monitor whether the SLO catches what the deleted alerts used to.