On-Call

Definition

On-call is the arrangement by which someone is responsible for responding to production issues outside working hours. Its design is an architectural concern because architecture largely determines how often it fires and how hard each page is to resolve.

The properties of a sustainable rotation

• Enough people. Fewer than six or seven makes the rotation punishing and drives attrition, which makes it smaller.
• A page rate people can live with. A widely used benchmark is no more than about two pages per shift; more than that and the responder cannot recover between them, and alert quality collapses because nobody has time to fix the causes.
• Every page is actionable. If the response is "acknowledge and go back to sleep", the alert is noise and must be deleted or downgraded. Tolerating one known-noisy alert is how a team learns to ignore alerts generally.
• The person paged can fix it. Access, permissions and knowledge. Paging someone who must then wake someone else has doubled the response time and achieved nothing.
• Time compensated, in money or in time off. Unpaid on-call is a tax on the same people repeatedly.
• Handover between shifts, so ongoing issues are not rediscovered.

The architectural connection

Teams that own what they build have a direct incentive to make it reliable — the feedback loop between a design decision and being woken by it is what "you build it, you run it" is actually for. Separating build from operate breaks that loop and produces systems that are locally convenient to write and expensive to run.

That connection cuts both ways, which is the useful part: a high page rate is architectural evidence. A service paging four times a week is telling you something about coupling, capacity headroom, dependency fragility or deployment safety, and treating it as a rota problem rather than a design problem is how it persists for years.

Failure scenarios

• Alert fatigue. So many pages that real ones are missed. The mechanism is well documented and the remedy is aggressive deletion, not better triage.
• A hero. One person who knows how everything works and is therefore always contacted. This is a single point of failure with a name and a family.
• No runbooks, so every page requires original thought at 3am.
• Pages for things that can wait. If it can be handled at 9am, it is a ticket.
• No follow-up loop, so the same page recurs indefinitely.

Trade-offs

A tighter rotation with fast response buys availability and costs people. That trade must be made explicitly and revisited, because its cost accumulates invisibly until someone resigns — at which point it is attributed to something else.

Interview question

"Your team is paged eleven times a week and morale is poor. What do you do first?"