Alert Fatigue in Practice

Definition

Alert fatigue is what happens when the rate of non-actionable alerts is high enough that responders stop treating alerts as significant. It is not a human weakness; it is the predictable and correct adaptation to a low signal-to-noise ratio.

The consequence is that the alert that mattered is missed — so a monitoring system past this threshold has made the service less reliable.

The test for whether an alert should exist

Does a human need to take action, now, that cannot be automated?

• No action needed → delete it.
• Action needed but not urgent → make it a ticket, not a page.
• Action needed and automatable → automate it, and alert only if the automation fails.
• Action needed now, by a human → keep it. This is a page.

Applied honestly, this deletes most alerts in a typical estate. That is not a loss of coverage; it is the removal of things that were never coverage.

Symptom-based rather than cause-based alerting

Alerting on causes — CPU high, disk filling, queue growing — produces many alerts, most of which do not affect users. Alerting on symptoms — the SLO is burning, checkout error rate is elevated, requests are timing out — produces few alerts, all of which matter.

Causes belong on dashboards and in tickets. They are what you look at after a symptom alert fires, to find out why.

The strongest form of this is error budget burn rate alerting: page when the budget is being consumed fast enough to be exhausted before the window ends. A fast burn pages immediately; a slow burn becomes a ticket. This automatically calibrates urgency to actual user impact, which is what static thresholds never do.

Practical measures

• Track alert volume per shift as a metric and treat exceeding the target as a defect with an owner.
• Review every alert that fired without action, weekly, and delete or downgrade it.
• Group related alerts. One incident should page once, not thirty times.
• Alert on the age of things, not their count. Queue depth means nothing without throughput; the age of the oldest unprocessed item is directly meaningful.
• Delete rather than tune. A noisy alert kept "because it might be useful one day" costs attention every week.

Failure scenarios

• The permanently-firing alert everyone has learned to ignore, which by definition provides no signal.
• Every service alerting independently during one incident, burying the causal one.
• Alerts with no runbook, requiring original thought at 3am.
• Thresholds set once and never revisited, so growth makes them noise.

Interview question

"Half your pages result in no action. Walk me through how you would fix that without reducing coverage."