A platform's alerts are defined per component - CPU, memory, queue depth, replica lag. During an incident, forty alerts fire simultaneously. What should the alerting strategy be instead?
Show the full answer Hide the answer
The problem
Cause-based alerts fire in swarms because one cause produces many symptoms. A slow database produces high queue depth in six services, elevated CPU in three, replica lag, connection pool exhaustion and timeout errors — forty alerts describing one problem, arriving at the worst possible moment.
The responder's first task becomes triage of the alerts rather than diagnosis of the system.
Worse, cause-based alerts are individually poor signals. High CPU is not a problem if users are unaffected; low CPU is not reassuring if they are.
What to do instead
1. Alert on symptoms, at the user-visible boundary. "Checkout success rate below threshold" is one alert that fires for any of the forty causes. It is actionable by definition, because if it fires, users are affected.
2. Base alerts on SLO burn rate. Alert when the error budget is being consumed fast enough to matter, with multiple windows: a fast burn (severe, page immediately) and a slow burn (a ticket). This makes urgency proportional to consequence, rather than to whether a threshold was crossed.
3. Demote cause-based signals to diagnostics. CPU, queue depth and replica lag remain on dashboards and in runbooks — as the things you look at after being paged, not the things that page you.
4. Keep a small number of cause-based alerts for conditions that are certain future problems even without current symptoms: disk approaching full, certificate approaching expiry, quota approaching a ceiling. These are predictions, not symptoms, and they justify their place.
5. Group and deduplicate, so related alerts collapse into one incident rather than forty notifications.
The test for every alert
If this fires at 3 a.m., is there something a human must do immediately?
If the answer is no, it is a dashboard or a ticket. Applying this test rigorously typically removes the majority of a mature system's alerts, and the removal improves reliability rather than reducing it — because the remaining alerts are trusted and acted on.
The measurement that drives improvement
Track every page by outcome: actionable and urgent, actionable but could have waited, or not actionable. In struggling rotations the third category dominates, and measuring it reframes the problem from "we need a bigger rotation" to "we need fewer alerts". That reframing is usually the whole fix.