A team has 340 alerts. On-call receives roughly 40 pages a week and most are ignored. What would you change?
Show the full answer Hide the answer
What the interviewer is testing
Whether you rebuild the alerting philosophy rather than tuning 340 thresholds.
The diagnosis
They are alerting on causes rather than symptoms. CPU, memory, disk, queue depth, individual host health — conditions that may or may not indicate a problem. Most fire when nothing is wrong, and a novel failure that genuinely breaks a user journey trips none of them.
40 pages a week with most ignored means the paging channel has been trained into noise. The system now has negative detection value, because a real page arrives in a stream of ones that were not.
What I would change
Delete almost all of them. Start from zero rather than tuning. This is the move that requires nerve and it is the correct one.
Page only on symptoms: error rate, latency, and the success of critical user journeys. These catch every impactful failure including ones nobody anticipated, and stay quiet when internal conditions are unusual but harmless.
Use burn-rate alerting against an error budget rather than static thresholds. A fast burn pages; a slow burn raises a ticket. One mechanism replaces a large number of thresholds and scales urgency to consequence.
Move cause-based signals to dashboards and tickets. They remain valuable for diagnosis once a symptom alert fires, and for genuinely predictive conditions like a certificate expiring or a disk filling within the day. They just do not wake anyone.
Every page must be actionable. If the response is "acknowledge and go back to sleep", it is not a page.
The target
Fewer than two pages per on-call shift, each requiring action. That is the number at which pages are believed and acted on.
What a strong answer adds
Reviewing pages as a standing practice: every page examined weekly for whether it was actionable, and non-actionable ones deleted or downgraded. Alert sets decay towards noise without this, because alerts are added after incidents and never removed.
And the measure that shows it working: time to detect, plus the proportion of incidents detected internally rather than reported by customers.
Common weak answers
Tuning the 340 thresholds. Routing alerts to different channels, which redistributes the noise.