Incidents at your company are chaotic: unclear ownership, no communication, and postmortems that produce nothing. Design the improvement.

Roles, so that coordination exists

The failure is everyone investigating and nobody coordinating. Ten people debugging is slower than three debugging and one directing.

Incident commander — coordinates, decides, does not debug. This is the discipline that makes it work and the hardest for a technical person to hold. Operations lead — makes the changes. Communications lead — updates stakeholders and the status page, shielding responders from interruption. Scribe — records the timeline as it happens.

Small incidents collapse these into one person. The structure exists so scaling up is a known move rather than an improvisation.

Two rules: whoever declares is commander until explicit handover, and seniority does not override — a junior engineer can command with a principal as operations lead.

Severity, so the response is predetermined

A small scale — SEV1 through SEV4 — defined by user impact, not by which component failed.

Each level must carry a predefined response: who is paged, who is notified, whether customers are told, how often updates go out. If it does not determine those, it is a label rather than a mechanism.

Declaring high must be free. Downgrading costs minutes; under-declaring costs an hour of the wrong response.

Process, so time is not lost to decisions

Declare early — the cost of standing down is far lower than a late declaration.

Mitigate before diagnosing. Restore service, then find out why. Teams routinely invert this.

A single channel per incident, with regular timed updates whether or not there is news, because silence generates interruptions.

Explicit handover with a stated summary for anything spanning shifts.

Postmortems that produce something

Blameless — for information, not kindness. In a blaming culture people withhold details, and a review built on incomplete information produces the wrong actions. The reframe that does the work: "why did this seem reasonable at the time?"

Contributing factors, not a root cause. Complex systems fail because several conditions were true at once. Stopping at the last human action is the least useful place to stop.

Measure the timeline gaps — failure to detection (a monitoring problem) and detection to mitigation (a response problem). Both are trackable over time and both point at concrete work.

Action items specific, owned, prioritised and tracked in the normal backlog. Untracked actions are the most common failure of the practice, and they teach the organisation that the exercise is ceremonial.

Sustainability, or none of it holds

At most two pages per shift, a rotation of six to eight, compensation, and protected follow-up time after a disrupted night. And the team that builds the service is on call for it — a feedback loop, not a punishment.

Prove it before you need it

Tabletop exercises first: an hour, no risk, and they reliably find undefined declaration authority, stale runbooks, missing permissions and escalation paths ending at someone who left. Then non-production simulation, then a controlled production game day.