An on-call engineer receives so many alerts that they routinely acknowledge without reading. What has failed, and what is the risk beyond tiredness?

What has failed

The signal-to-noise ratio has crossed the point where reading is no longer rational. If nineteen of twenty alerts require no action, the expected value of investigating any given alert is low, and acknowledging without reading is an efficient adaptation — not carelessness.

That framing matters, because organisations tend to treat this as an individual discipline problem, which guarantees it will not be fixed.

The risk beyond tiredness

The one alert that mattered is acknowledged without being read. This is not hypothetical; it is the predictable outcome, and it is how significant incidents begin with a signal that fired correctly and was dismissed.

Second-order risks compound:

• Detection time lengthens across all incidents, since alerts stop being the detection mechanism and customer reports take over.
• Alert thresholds get loosened to reduce volume, which reduces sensitivity to real problems.
• Alerts get muted, sometimes permanently, and the mute outlives the reason for it.
• Trust in monitoring collapses, so during a real incident engineers do not believe the dashboards and waste time re-verifying.
• Turnover rises, and departures take system knowledge, lengthening future incidents.

What to change

1. Measure actionability, splitting pages into actionable-and-urgent, actionable-but-not-urgent, and not-actionable. Publish the numbers.

2. Delete the non-actionable, aggressively. Not tune — delete. A tuned threshold on a signal nobody acts on is still noise.

3. Alert on symptoms rather than causes, which collapses alert swarms into single incidents.

4. Fix the top recurring causes. Rank alerts by count; the top few usually dominate. Recurring pages are unfinished work being paid for in sleep.

5. Automate the deterministic responses. Any alert whose runbook is "restart it" or "scale it" should be automated — if a human is following a fixed procedure, a machine should.

6. Require an owner and a runbook for every alert, and delete those that have neither. An alert nobody owns will not be acted on regardless.

The cultural requirement

Deleting alerts must be safe and encouraged. In many organisations removing an alert feels risky — what if it would have caught something? — so alerts only accumulate. Making deletion a normal, reviewed activity, with the volume reduction treated as a reliability improvement rather than a coverage regression, is what breaks the accumulation.