intermediate 2 min answer

A platform's on-call rotation is causing burnout - engineers are paged frequently at night and turnover is rising. What should be measured and changed?

oncallalert-qualitysustainabilitytoilatlassiandebugging
Show the full answer Hide the answer

What to measure first

Pages per shift, broken down by actionability. The critical distinction:

  • Actionable and urgent — required a human, now. This is what on-call exists for.
  • Actionable but not urgent — needed doing, could have waited until morning. Should not have paged.
  • Not actionable — the alert fired, the engineer looked, nothing was wrong or nothing could be done. Pure cost.

In most struggling rotations, the third category dominates. Measuring it usually surprises people and immediately reframes the problem from "we need more people on the rotation" to "we need fewer alerts".

Then: pages per night specifically; time to resolution; repeat pages for the same cause; and the proportion of the shift spent on toil versus genuine incidents.

What to change

1. Delete or downgrade non-actionable alerts, aggressively. The bar is: if this fires at 3 a.m., is there something a human must do immediately? If not, it is a ticket or a dashboard, not a page. This single change typically removes the majority of night pages.

2. Alert on symptoms, not causes. "Checkout success rate below threshold" is one actionable alert. "CPU high", "queue deep", "replica lagging" are twenty alerts that fire together during one incident, producing a storm at the worst moment.

3. Fix the top recurring causes. Rank by page count. The top three causes usually account for most night pages, and fixing them properly is a better investment than any rotation change. Recurring pages are unfinished work being paid for in sleep.

4. Automate the known responses. Any page whose runbook is "restart it" or "scale it up" should be automated. If a human is following a deterministic procedure, a machine should follow it.

5. Make follow-up work count. Time spent fixing the causes of pages must be visible and valued in planning, or the rotation stays broken because fixing it is invisible work.

6. Rotation size and compensation. Enough people that the shift is infrequent, and explicit recognition of the burden. This is a mitigation, not a fix — a bad rotation spread over more people is still a bad rotation.

The organisational signal

Rising turnover among on-call engineers is a reliability signal, not just an HR one. People leaving takes system knowledge with them, which lengthens future incidents, which increases on-call burden — a reinforcing loop that gets worse on its own.

The principle worth stating

On-call load is a measure of system quality. A system that pages its owners nightly is telling you it is not finished. Treating the pain as a staffing problem rather than an engineering signal is how organisations normalise a system that should have been fixed.