practice

Runbook

also called Playbook

A short, actionable document telling an on-call engineer what an alert means, what to check, and what the safe mitigations are.

operationsoncalldocumentation

The test of a runbook is whether someone who did not build the system can act on it at 3 AM. That rules out architecture overviews, prose explanations and links to a design document.

What belongs in one: what the alert means in terms of user impact, the first three things to check with the exact queries or dashboard links, the known causes and their mitigations, the safe actions (restart this, fail over that, disable this flag) and explicitly the unsafe ones, the escalation contact, and how to tell when it is resolved.

Two rules keep them alive. Every page links to its runbook — an alert without one is a puzzle handed to a tired person. And the runbook is updated during the incident that revealed it was wrong, not afterwards, because afterwards does not happen.

The natural progression is that a runbook step performed repeatedly becomes automation, and the runbook then documents how to trigger and verify it. A runbook full of manual steps that are always the same is a backlog item.