concept

Alert Fatigue

The desensitisation that follows from alerts that are frequent, non-actionable, or not tied to user impact — after which real alerts are missed too.

alertingoncallnoise

The failure is predictable. Alerts are added after each incident, each seems reasonable in isolation, thresholds are set on causes rather than symptoms, and within a year the on-call rota receives forty notifications a night and acknowledges them reflexively.

The corrective principle is that every page must be urgent, actionable and user-impacting. If a human cannot do something useful about it right now, it is not a page — it is a ticket, a dashboard, or nothing.

Alerting on SLO burn rate rather than on component thresholds is the structural fix: it fires when users are actually being affected at a rate that will exhaust the budget, which collapses a hundred cause-based alerts into a handful of symptom-based ones.