The Base Rate Problem in Alerting
Why a detector with excellent sensitivity and specificity still produces mostly false alarms, the arithmetic that shows it, and the design responses that make an alerting system usable.
An anomaly detector is reported as 99 percent accurate and the on-call engineer stops reading its alerts within a month. Both facts are consistent, and the arithmetic connecting them is the single most important thing to understand about deploying detection.
The arithmetic
Suppose a detector runs every minute on a metric, so 1,440 evaluations per day. Suppose genuine anomalies occur once a week. Suppose the detector catches 95 percent of them and has a 1 percent false positive rate.
Per day: 14.4 false alarms, and 0.14 true alarms. Precision is under 1 percent. The engineer sees roughly a hundred alerts a week, one of which matters, and the rational response is to ignore the channel.
The problem is not the detector's quality. A 1 percent false positive rate is a good detector by any conventional measure. The problem is that the evaluation count is enormous and the event rate is tiny, so even a small per-evaluation error rate overwhelms the signal.
What actually helps
Reduce the number of tests. Alert on a small number of metrics that matter rather than on everything measurable. A hundred monitored metrics is a hundred times the false alarm rate, and most of them nobody would act on anyway.
Require persistence. Demanding that a condition hold for several consecutive intervals cuts independent false positives multiplicatively while delaying detection linearly. A condition that must persist for five minutes has a false positive rate roughly the fifth power of the per-minute rate, which is usually the single most effective change available.
Aggregate correlated alerts. One upstream failure triggering forty downstream detectors should produce one alert naming the root cause. Without correlation, the alert volume during an incident is at its highest exactly when attention is scarcest.
Route by consequence. Only conditions requiring immediate human action should page. Everything else belongs on a dashboard or in a daily digest, and the distinction should be made per alert rather than by severity label.
Set thresholds from the acceptable alert budget. Deciding how many false alarms per week are tolerable, then setting the threshold to deliver that, is the design that survives contact with the on-call rotation. Setting the threshold from statistical significance and discovering the volume afterwards does not.
When it breaks
Suppression hides real problems. Every measure above trades sensitivity for precision, so a system tuned to be quiet misses things. That is the correct trade when the alternative is a channel nobody reads, and it should be an explicit decision with a stated miss rate rather than an accident of tuning.
Alert fatigue is not reversible by fixing the detector. Once a channel has been ignored for months, improving its precision does not restore attention automatically. Re-establishing trust requires a visible reset, and it is much easier to avoid than to repair.
Fixed thresholds fail on seasonal data. A threshold set from one week fires every weekend and every holiday. Seasonal baselines are not an enhancement here, they are what makes a threshold meaningful at all.
The metric with the most alerts is not the most important. Alert volume reflects variance and threshold placement, not consequence, and systems that prioritise by volume systematically direct attention to the noisiest rather than the most critical.
12 flashcards for this concept
Click a card to reveal the answer.