You join a team of 12 engineers taking around 40 pages a week, of which perhaps 5 required action. The team is exhausted and two people have resigned. The director asks for a plan. What do you do, and in what order?
Show the full answer Hide the answer
What the interviewer is testing
Whether you treat this as an operational problem with a measurable target, or as a morale problem to be solved with sympathy and a rota change. The number is the finding: 40 pages with 5 actionable means 88% of the pages are noise, and noise has a cost that compounds - it trains people to acknowledge without reading, which is how the one real page gets missed.
The clarifying questions
- What fraction of pages correspond to user-visible harm? This separates alerts on symptoms from alerts on causes, and it is the axis that matters.
- How many alerts have a runbook, and how many runbooks have been read this quarter?
- What is the current target? If there is no target for pages per shift, there is nothing to manage against.
- Which services generate the pages? In most teams the distribution is severe: a small number of services generate the large majority.
The plan, in order
1. Set the target explicitly. No more than one or two pages per 12-hour shift, which is the widely used figure from Google's SRE practice and is chosen so that a responder can actually investigate each one. Declare it and measure against it weekly.
2. Delete, do not tune, first. Any alert that has fired more than a few times in a quarter without producing an action is deleted or downgraded to a ticket. This is fast, requires no engineering, and typically removes most of the volume. Expect resistance of the form "but what if"; the answer is that an alert nobody acts on is already not being acted on.
3. Move cause alerts to symptom alerts. Page on the user-visible thing - error rate, latency, a failing journey - not on CPU, disk or a queue length that may be harmless. One symptom alert replaces a dozen cause alerts and it fires when something is actually wrong.
4. Fix the top generator. The service producing the most pages gets a week of engineering time, explicitly prioritised over feature work, with the page count as the acceptance criterion. Repeat.
5. Route the non-urgent to a queue with an owner and an SLA. Much of what pages at night is real work that does not need doing at night.
6. Make the on-call load visible to the people who allocate work. Pages per week per service, on the same dashboard as delivery metrics, with the resignations named as the cost. An error-budget policy formalises this: when reliability work is not done, feature work stops.
Common weak answers
- Rotating more people through the rota, which spreads the damage rather than reducing it and burns the new people too.
- Tuning thresholds one at a time, which is slow and preserves the belief that every alert deserves to exist.
- Buying a tool. Noise reduction and grouping make the symptom bearable and remove the pressure that would have fixed the cause.
- Treating it as morale. Pizza and sympathy for a system that wakes people twice a night is an insult with a budget.
What a strong answer adds
The organisational half: this state is produced by a prioritisation system that rewards features and absorbs reliability work silently, so fixing the alerts without changing that returns the team to 40 pages within a year. And the honest sequencing note - the first two steps cost almost nothing and produce most of the improvement, so there is no reason for anyone to wait for a quarter of planning.
When the rota is the real problem
If pages are genuinely few and genuinely urgent but concentrated on two people, then this is a rota and staffing question after all. Check the distribution before diagnosing the alerts: a median of two pages a week with one engineer taking twenty is a different problem with a different fix.