Operational Runbook

flowchart TB
  s(["Alert: order submit p99 > 900 ms<br/>for 5 min"])
  s --> c1["1. Confirm scope<br/><i>one region or global?</i>"]
  c1 --> q1{"Global?"}
  q1 -->|"no"| a1["2a. Check that region's<br/>dependency health"]
  q1 -->|"yes"| a2["2b. Check shared dependencies:<br/>DB, cache, identity"]
  a1 --> q2{"Region-local cause?"}
  q2 -->|"yes"| m1["3. Shift traffic away<br/><i>runbook OPS-04</i>"]
  q2 -->|"no"| a2
  a2 --> q3{"DB CPU > 80%<br/>or replication lag > 10s?"}
  q3 -->|"yes"| m2["4. Enable read-path cache bypass=off<br/>+ raise connection limit<br/><i>flag: cache.aggressive</i>"]
  q3 -->|"no"| q4{"Recent deploy<br/>in last 60 min?"}
  q4 -->|"yes"| m3["5. Roll back<br/><i>runbook OPS-01</i>"]
  q4 -->|"no"| esc(["6. Escalate to<br/>Order team on-call<br/><i>stop investigating</i>"])
  m1 --> v["7. Verify p99 recovers<br/>within 10 min"]
  m2 --> v
  m3 --> v
  v --> q5{"Recovered?"}
  q5 -->|"yes"| done(["Close · record in incident log"])
  q5 -->|"no"| esc

What it is

One alert, one procedure. Not a wiki page about the system — a decision tree an engineer who did not build it can execute at four in the morning without judgement calls they are not equipped to make.

The most important box is the escalation terminal. A runbook that never says "stop and escalate" invites an unfamiliar engineer to keep trying things during an incident, which is how a degradation becomes an outage.

When you produce it

With every alert. An alert without a runbook is a page with no instructions, and it is the fastest route to alert fatigue. Update it after every incident where it did not match reality.

Who reads it

Whoever is on call, including people from outside the owning team. It is written for the least familiar plausible reader.

What good looks like

• It is attached to the alert, so it opens from the notification.
• Every mitigation names the exact command, flag or linked runbook — no "restart the service as appropriate".
• Verification step with a time bound, so success is defined.
• A clear escalation point with a named rota.
• Tested in a game day. An untested runbook is a hypothesis.
• Dated and owned, and reviewed when the system changes.

Common mistakes

• A page of background instead of a procedure.
• No escalation, so there is no defined end to a failed attempt.
• Steps that assume tribal knowledge — which dashboard, which flag, which account.
• Never exercised, so the first real use finds the broken link.