A team receives hundreds of alerts a day and misses a genuine outage. What is the fastest structural fix, and what should replace the current approach?

Why the alternatives do not work

Tuning thresholds reduces volume marginally and keeps the structural problem: alerts on causes fire in clusters during a single incident, so one event produces forty pages regardless of threshold. More people spreads the load without reducing it, and adds coordination cost. Anomaly detection on top adds a new source of alerts with worse explainability, on an input signal that is already noise.

The structural fix

Alert on symptoms the user experiences; investigate causes with dashboards.

One page saying "trip assignment success rate below threshold" is worth more than forty saying individual hosts have high CPU, a queue is deep, a connection pool is saturated and a cache hit rate has dropped. Those forty are the investigation, and they belong on a dashboard the responder opens after being paged once.

The test for every existing alert: if this fires at 3am, is there something a human must do right now? If not, it is a dashboard entry or a ticket. Applying this honestly typically removes most of the volume.

The specific pathologies to remove

• Alerts on individual instances in a fleet where instance loss is handled automatically.
• Alerts on resource utilisation rather than on the effect of it. High CPU is only a problem if something is degraded.
• Alerts that have never resulted in an action, which is measurable and should be reviewed monthly.
• Alerts duplicated across layers — the same failure detected by an infrastructure monitor, an application monitor and a synthetic check, producing three pages.
• Alerts kept because deleting them feels risky, which is the reason most of them survive.

The measurement that drives improvement

Percentage of pages that resulted in an action, and pages per shift against a number the team has agreed is sustainable. Both are simple, both are rarely tracked, and a team that starts tracking them almost always discovers that a small number of sources produce most of the load — which makes the fix specific rather than a general clean-up.

The framing

A noisy pager is an architectural signal, not a monitoring configuration problem. Persistent alerting on the same class of failure means the system requires human intervention to stay up, and the durable fix is to remove the need rather than to page more sustainably.