Symptom-Based Alerting
Alerting on degradation the user experiences rather than on the internal conditions that might cause it.
Cause-based alerts — high CPU, low disk, a full queue, an unhealthy instance — are noisy in both directions. They fire when nothing is wrong (CPU is high because work is being done) and stay silent when something is (users are failing for a reason nobody wrote a rule for).
Symptom-based alerts fire on elevated error rate, latency beyond the SLO threshold, or throughput collapse — conditions that by definition mean users are affected. They catch causes nobody anticipated, which is most of them.
The four golden signals — latency, traffic, errors, saturation — are the standard starting set, and the first three are symptoms.
Causes are still worth monitoring; they belong on dashboards for diagnosis and as tickets for capacity, not as pages. Disk filling in six days is real and important and should never wake anyone.
The refinement that makes it operational: alert on symptoms with burn-rate windows so severity follows the size of the impact, and attach to each alert a runbook and the dashboard that shows the likely causes — so the page says what is wrong for the user, and the linked context answers why.