An on-call team receives 60 alerts per shift and has stopped reading most of them. What is the structural fix, and what should the alerting rules actually be based on?

The situation is worse than it appears

At 60 alerts per shift, the team has rationally learned to ignore alerts, and that adaptation applies to the real ones too. The alerting system now has a negative value: it consumes attention, degrades the people who would otherwise respond, and provides no reliable signal.

Adding a better alert to this system changes nothing, because it arrives in a channel nobody reads.

The rule that fixes most of it

Page a human only when a human must act immediately. Everything else is a ticket, a dashboard, or a scheduled report.

Applying this honestly eliminates the majority of alerts, because most are informational ("CPU is 80%"), automatically resolving ("a pod restarted"), or not actionable ("disk on a stateless node is 70% full").

Alert on symptoms, not causes

Alert on what the user experiences: error rate, latency, and whether the critical journeys succeed.

Cause-based alerts — high CPU, high memory, a full queue, a slow query — fire when the system is fine and stay silent when it is not. High CPU may be an efficient system under load; a healthy CPU is entirely compatible with a total outage caused by a dependency.

The exception is predictive alerts on genuinely irreversible conditions: a disk that will fill in four hours, a certificate expiring in a week, a quota approaching its limit. These are cause-based and legitimate, because by the time the symptom appears it is too late to act.

Alert on burn rate, not on thresholds

An SLO-based alert with multi-window burn rate is the strongest general mechanism: a fast burn over a short window pages immediately; a slow burn over a long window opens a ticket. This automatically distinguishes "we are having an outage now" from "we are gradually consuming our budget", which threshold alerts cannot do and which is the distinction that determines the correct response.

The mechanical reductions

• Delete every alert that has not been acted on in 90 days. If nobody acted, it is not an alert. This single action typically removes half of them, and the objection "but we might need it" is answered by the fact that it fired and nobody responded.
• Group and deduplicate: one incident should produce one page, not forty. Alerts from the same root cause are correlated by time and dependency and should arrive as one notification.
• Suppress downstream alerts when an upstream dependency is already alerting.
• Ensure every alert has a runbook. An alert with no documented response is an interruption with a question attached; the runbook requirement also naturally deletes alerts, because writing one exposes that there is no action.
• Distinguish severity properly and route accordingly — page, ticket, or dashboard only.
• Auto-resolve anything that recovers on its own.

The governance that keeps it fixed

Review alerts in every post-mortem: which fired, which were noise, which were missing. And track pages per shift as a metric with a target — a team receiving more than a couple of pages per shift is not on-call, it is performing manual operations, and the response is engineering rather than tolerance.