The same class of incident has recurred four times in a year despite each postmortem producing actions that were completed. What is going wrong?
Show the full answer Hide the answer
The actions addressed the trigger, not the system
Four incidents with four completed remediations and a fifth incident means the remediations were local: fix this timeout, add this alert, patch this service. Each was correct and none changed the conditions that keep producing the class.
Look for the reinforcing loop
Recurring failures usually sit inside a feedback structure. Common ones:
Load without backpressure. A slow dependency causes retries, retries increase load, load increases slowness. Each incident is fixed by scaling; the loop remains.
Reliability work deprioritised by delivery pressure, which produces incidents, which consume the capacity that would have done the reliability work.
Knowledge concentration. One person handles the incidents, so nobody else learns, so that person handles the next one.
Coupling nobody owns. A shared component whose failure affects everyone and whose maintenance is on nobody's roadmap.
Ask what the four have in common, not what caused each
Put the four postmortems side by side and look for the shared condition rather than the shared trigger. The answer is frequently structural: a shared database, a missing bulkhead, an alert routed to a team without authority to act, a dependency with no timeout budget.
Then change the structure
Bulkheads so the failure cannot spread. A retry budget so the amplification loop cannot form. A named owner with allocated capacity for the shared component. Changing who is paged so the team that can fix it is the team that feels it.
The postmortem practice to change
Add a standing question: has this class occurred before, and what did we do then? If the answer is yes and the remediation was local, the action for this one is structural — and it needs a sponsor, because it will be larger than a sprint and will not fit in the incident's follow-up ticket.