Alert on Symptoms

Cause-based alerting produces the two failure modes that destroy trust in a paging system simultaneously: alerts that fire when nothing is wrong, and outages that produce no alert because the cause was one nobody had thought to monitor.

A disk at 85%, CPU at 90%, a node restarting, a queue with 10,000 messages — none of these is necessarily a problem, and each will page someone at 3 AM to look at a healthy system. Meanwhile a novel failure that makes checkout fail for 30% of users may trip none of them.

Symptom-based alerting inverts it: page on error rate, latency, and the failure of critical user journeys — the things that mean a customer is having a bad time. This alerts on every impactful failure including ones never anticipated, and stays quiet when internal conditions are unusual but harmless.

The refinement that makes it work at scale is error budget burn rate: alerting when the rate of budget consumption implies the objective will be missed, with a fast burn paging immediately and a slow burn raising a ticket. That single mechanism replaces a large number of static thresholds and scales the urgency to the consequence.

Cause-based signals keep their place — as dashboards and tickets, for diagnosis once a symptom alert fires, and for genuinely predictive conditions such as a certificate expiring or a disk that will fill within the day. The distinction is what pages a human at night.