The three largest cloud outages of 2025 were each prevented by a design rule that had been public, codified and enforced in open-source contributor documentation for six years. This guide reconstructs the machinery organisations put between a production failure and the next design decision, from the published post-incident processes of Google, Amazon, GitLab, Etsy, Kubernetes and Envoy, and from six first-party incident reports, and it identifies the two stages of that machinery where the evidence says the signal is lost.
The problem, stated without naming the practice: an expensive surprise produces a document, and nobody can show that the document changed the system.
Start with the sentence that motivated this guide, because it is the whole argument in one line. On 2 July 2019, after a regular expression took down eighty percent of its traffic for twenty-seven minutes, Cloudflare committed publicly to "changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare". On 18 November 2025, a Bot Management feature file doubled in size, propagated globally in minutes, exceeded a hardcoded limit and stopped the majority of core traffic through the Cloudflare network for three hours and forty-six minutes. The remediation Cloudflare published that evening was to begin "hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input". Both incidents are the same shape: an artefact that changes production behaviour, generated inside the company, distributed globally in minutes, and never routed through the release machinery the company had already built. The 2019 lesson was correct. It was applied to WAF rules, which is where it had hurt, and not to the property that made WAF rules dangerous.
That is not a Cloudflare story. In the same eighteen months, Google Cloud lost Service Control globally because a quota-policy code path shipped without a feature flag and a policy row with blank fields dereferenced a null pointer in every region at once; AWS lost the DynamoDB regional endpoint in us-east-1 for the better part of a day because a latent race condition between two instances of the automation that manages DNS deleted every address record; and CrowdStrike pushed a content file whose parameter count did not match the sensor that read it, to every machine, with no staged deployment, because content was not code. Four organisations, four different stacks, one class of failure. Every one of them had a post-incident process, and every one of those processes is documented in public.
An architect who is being asked to stand up or defend a post-incident process, and who needs to know which parts of it are load-bearing, which parts the published evidence does not support, and what to build instead of the parts that do not work.
Scope. This guide covers the machinery between a production failure and a durable change to the system: what triggers an analysis, what the analysis produces, who approves it, how the lesson is encoded, and where the signal is lost. It deliberately does not cover incident response itself, on-call design, paging and alerting, severity taxonomies, status page communications, chaos engineering, or the security-incident and regulatory-disclosure path, which has different drivers and a different audience. It also does not cover the human cost of being on the sharp end of an incident, which is real, well documented elsewhere, and not an architecture question.
A note on confidence, because it varies sharply across this topic. Incident mechanism is reported: the first-party postmortems in section 4 are specific, timestamped and internally consistent. Process design is reported but unverified: six organisations describe what their process is, and none describes what it produced. Process outcome is open: no organisation has published a completion rate for corrective actions, a repeat-incident rate, or a before-and-after measurement. Where this guide says a mechanism works, that claim is a reconstruction from incident evidence, and it is marked.
The common shape across Google, Amazon, GitLab, Etsy, Kubernetes and Envoy, with the divergence points marked and attributed.
Lay the six published processes side by side and the first four stages are present in all of them. Something triggers an analysis. A timeline is reconstructed. An artefact is written. Actions are recorded in whatever tracker the organisation already uses. The differences at those stages are real but small: Google requires that "all postmortems which follow a user-affecting outage must have at least one P[01] bug associated with them", GitLab requires the incident lead to open a review for every Severity 1 and Severity 2 and closes it within five working days, and Amazon routes serious Correction of Error documents up a management chain to a director or a chief technology officer. Those are variations in cadence and audience, not in structure.
The interesting divergence is at stage five and stage six, and it is where the whole difference in outcome sits.
Stage five is generalisation: asking whether the lesson applies anywhere other than the component that broke. Only one of the six processes asks it as a written question. GitLab's handbook puts it in the list every review owner is expected to answer and every Bar Raiser is expected to probe: "Are there areas outside of this specific impact that need to be addressed in the same fashion?" That single sentence is the difference between Cloudflare's 2019 remediation list, which fixed WAF rule rollout, and a remediation list that would have found the Bot Management feature file six years early. Neither Cloudflare postmortem mentions such a question, and no public source says whether one was asked internally; this is a reconstruction from what the two remediation lists contain.
Stage six is encoding: turning the lesson into something mechanical that constrains future changes made by people who were not in the room. Only two of the six do this structurally, and neither of them is an incident-response process. Envoy requires that high-risk and user-visible behavioural changes ship behind a runtime guard that is safe to flip on a running process, a rule a maintainer wrote into the contributor contract as documentation plus tooling in March 2019. Kubernetes requires every feature to pass a Production Readiness Review whose stated purpose is to ensure features "can be disabled or rolled back in the event they cause increased failures in production", and whose questionnaire opens with Feature Enablement and Rollback and asks, mechanically, whether the feature can be disabled once enabled, how a rollback can fail, and what specific metrics should inform one.
Put those two facts next to Google's June 2025 remediation, which reads: "Enforce all changes to critical binaries to be feature flag protected and disabled by default." That is the Envoy rule of 2019 and the Kubernetes gate of 2019, adopted by a Google control plane in 2025 after a global outage. The knowledge was not missing. The gate was missing.
Every process triggers on customer-visible impact. None of them triggers on a near miss, and the VOID corpus shows the consequence: near-miss reports are close to absent from the public record, which Nash attributes to difficulty of detection, no process engagement, and reluctance to publish. The events with the best learning-to-damage ratio are the ones nothing catches.
Three different answers to the same problem, which is that the team that owns the service is the worst-placed group to judge whether its own analysis was deep enough. GitLab appoints a Bar Raiser panel of peers who must approve before a review can close. Amazon escalates to leadership. Etsy trains a facilitator whose job is the questions, not the conclusions. The Bar Raiser is the cheapest of the three to copy.
A question asked automatically of every future change, in the place where changes are made. Envoy puts it in the contributor contract and enforces it with tooling; Kubernetes puts it in a questionnaire whose answers are checked against machine-readable metadata. Neither is an incident process, which is exactly why both survive staff turnover and neither depends on anyone remembering the outage.
Runs this way at: Envoy, Kubernetes.
Four organisations described the same thing in four different vocabularies. CrowdStrike calls it Rapid Response Content. Cloudflare calls it a feature file. Google calls it a policy change replicated globally within seconds. AWS calls it a DNS plan. Because nobody has a shared name for it, nobody audits for it, so this guide will name it: the unpipelined change surface is every artefact that alters production behaviour without traversing the release pipeline the organisation built after its last outage. Configuration, feature files, policy rows, ML model weights, feature-flag values, rule content, DNS plans, index definitions and threat signatures all qualify. The defining property is not the file format. It is that the artefact reaches production faster than code does, and by a different route, and therefore skips staged rollout, canary analysis, rollback tooling and the review that would have asked what happens when it is malformed.
The transfer for an architect is direct and takes an afternoon: enumerate every path by which behaviour reaches your production systems, then mark each one with whether it is staged, whether it can be rolled back without a deploy, and whether the consumer validates the artefact as untrusted input. The paths that fail all three are where your next outage is. This is a reconstruction rather than a reported practice; no source describes an organisation running this audit, which is itself worth noticing given that four of them wrote the postmortem afterwards.
The public disagreements are sharper here than in most architecture topics, because two credible camps have been arguing in print for a decade.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Meeting output | Remediation list | Understanding only | Action is legible to management; understanding is not | ACM Queue, 2019 |
| Causal framing | Root cause, 5 Whys | Contributing conditions | Contracts and status pages demand a named cause | GitLab, 2017 |
| Lesson encoding | Procedure or runbook | Mechanical gate | Cheapest to write, easiest to mark done | Uptime Institute, 2025 |
| Closure authority | Independent Bar Raiser | Team self-closure | The owning team cannot judge its own depth | GitLab handbook |
| Process metric | Mean time to resolve | Coordination cost, SLOs | Duration is easy to collect and uncorrelated with severity | VOID via InfoQ, 2023 |
| Disclosure | Internal only | Public first-party account | Legal caution; 60% of public records name no cause | SoCC '16 |
Grouped by how the machinery broke rather than by company, because the same four classes account for every incident found in this hunt.
"A manager who complains that too few action items were produced has revealed his/her real interest: the reduction of an incident to a manageable discrete list of things to be done." Richard I. Cook, Adaptive Capacity Labs, June 2019
Every row carries its source and its date. The second table is the more useful one, because it is the list of things your board will ask for and you cannot get.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Catastrophic failures from bad error handling | 92% | 5 open-source distributed systems | 198 randomly sampled user-reported failures | 2014 | OSDI '14 |
| Of those, detectable by simple error-path testing | 58% | same corpus | no design knowledge required | 2014 | OSDI '14 |
| Public outage records naming no cause | 355 of 597 | 32 internet services | 1,247 news and postmortem links, 2009 to 2015 | 2016 | SoCC '16 |
| Public outage records naming the fix | 24% | same corpus | what was actually changed afterwards | 2016 | SoCC '16 |
| Practitioners defining the review by its remediation list | 91% | industry survey | core purpose of the post-incident meeting | 2019 | ACM Queue |
| Incidents resolved within two hours | 53% | VOID corpus | positively skewed, so the mean misleads | 2022 | InfoQ / VOID |
| Correlation between incident duration and severity | none detected | VOID corpus | roughly 2,000 reports, about 600 organisations | 2023 | InfoQ / VOID |
| Human-error outages tracing to procedures | 85% | Uptime Institute survey | ignored procedures or flawed procedures | 2025 | Uptime Institute |
| Organisations with a major human-error outage in 3 years | ~40% | same survey | self-reported | 2025 | Uptime Institute |
| Detection to correct diagnosis, Cloudflare | 2h 17m | 18 Nov 2025 | impact 11:20 UTC, cause identified 13:37 UTC | 2025 | Cloudflare |
| Total impact, AWS us-east-1 DynamoDB DNS | 15h 52m | 19 to 20 Oct 2025 | manual intervention required | 2025 | AWS |
| Gap between shipping the code and triggering it, Google | 14 days | 29 May to 12 Jun 2025 | path unexercised until a policy row arrived | 2025 | Google Cloud |
| Incident review closure target | 5 working days | GitLab | public RCA within 7 days for Severity 1 | 2026 | GitLab handbook |
| Time a change to the Kubernetes PRR gate waited on approval | 97 days | Kubernetes | then closed unmerged as a duplicate | 2026 | PR 5645 |
| Age of an open issue on a post-incident safety control | 4 years | Envoy | silent no-op on removed runtime guards | 2026 | Issue 20212 |
Measured: the OSDI and SoCC percentages, and every duration in this table, which comes from a first-party incident report with timestamps. Claimed: the Uptime Institute figures are self-reported survey responses from data-centre operators, not observations, and the sample skews toward facilities rather than software. Derived: the Cloudflare detection gap of 2h 17m and the Google gap of 14 days are subtractions from published timestamps, shown here so the arithmetic is visible. Stale risk: the VOID corpus figures are from 2022 and 2023 and the corpus has grown since; re-check before quoting the absolute counts, though the no-correlation finding is a property of the data rather than its size.
This is the most useful paragraph in the guide for anyone about to promise a board that a post-incident process will reduce incidents. Across thirty-four primary sources, six documented processes and six first-party incident reports, no organisation has published: a completion rate for post-incident corrective actions; a repeat-incident rate defined consistently and tracked over time; a before-and-after comparison of incident rate against the introduction of a review process; or the age distribution of open corrective actions. Google's SRE Workbook offers one anecdote, that action items from an earlier postmortem reduced the blast radius of a similar incident three years later. That is one data point, self-reported, with no denominator.
Figures do circulate. A completion rate of 30% and a repeat rate falling from 45% to 12% both appear on aggregator blogs and vendor pages. Neither cites a primary source and neither is used here. If you need the number, the only honest route is to instrument your own tracker: corrective actions opened, closed, and closed as "will not do", bucketed by age. It takes a query, and it is the single highest-value measurement in this entire topic because nobody else has it either.
Allspaw frames the missing denominator more sharply than any dashboard does. Counting incidents without counting the comparable occasions that did not become incidents tells you nothing: "Oh, you've had six incidents this week. Out of how many?" Nobody has published that denominator either.
Every source behind this page, graded and linked, with what to carry forward from each. Filter by kind.
The most detailed first-party account of an unpipelined-artefact failure available. Carries a minute-by-minute timeline including the two hours spent on a DDoS hypothesis, and names the hardcoded limit and the unwrapped error explicitly.
Seven numbered commitments, several with dates and completion status. Read it next to the 2025 report: the fifth commitment is staged rollout of rule content, and the 2025 incident is the same property in a different file.
A rare public account of a race condition inside operational automation, with the staleness-check mechanism described precisely enough to reimplement the bug. Also documents the decision to disable the automation globally before fixing it.
States plainly that the failing code path was never exercised during the region-by-region rollout because it needed a policy change to trigger it, and that the feature was not flag protected. The remediation list is the clearest statement of the rule in this guide.
An externally reviewed RCA with a numbered mitigation list. The sixth mitigation, staged deployment for Template Instances, is an admission that content had a separate and weaker release path than code.
Still the most complete public account of a recovery chain failing at every link, written while the company was live-streaming the recovery. Also a worked example of 5 Whys applied to two separate problems, the outage and the restore.
The rule as a contributor contract rather than an aspiration: high-risk and user-visible behavioural changes ship behind a named runtime flag that must remain safe to flip on a running process, with a deprecation timeline for removing the old path.
The change that turned a convention into a rule, authored by a maintainer, shipping documentation and tooling together. The review discussion is about ergonomics and test parameterisation, which is what adoption arguments actually look like.
Open for over four years. Documents the silent-failure mode of the safety control itself: when a guard is removed, an operator's override becomes a no-op with only a log line to say so.
Ninety-seven days waiting for a production-readiness approver, a stale-bot label, then closed as a duplicate. The clearest available evidence of what it costs to run a review gate that a small group must staff.
An open-sourced facilitation manual that instructs the facilitator to park remediation ideas until the timeline is complete, and states outright that prevention is not the goal. The most concrete artefact of the learning-over-fixing position.
The motivation is that feature authors and their own special interest group set the graduation criteria, so operator concerns went unrepresented. The remedy is a gate staffed by people deliberately outside the group.
Roughly thirty questions, opening with feature enablement and rollback, then observability, dependencies, scalability and troubleshooting. Answers are cross-checked against machine-readable metadata, so the gate is partly automated.
How the gate is actually run: who the approvers are, when in the release cycle the review happens, and the finding from the dry runs that a reviewer unfamiliar with the design catches items the owning group misses.
Three named roles, a Bar Raiser panel that must approve closure, a five working day target, and a written list of probing questions including the generalisation question that no other published process asks.
Survey data across data-centre and IT operators. The finding that matters here is that the overwhelming majority of human-error outages trace to ignored or inadequate procedures, which is the remediation category postmortems produce most.
Richard Cook on what people actually learn in a review, which is mostly about the organisation: which topics are discussable, how power works, and what management really wants from the meeting.
Allspaw's argument that the return on understanding the event exceeds the return on enumerating fixes for the components involved in it. Short, and the clearest statement of the position that opposes the Google and Amazon default.
Hochstein argues from scarcity: action items get planning time regardless, the cross-organisational hour does not recur, and he is explicit that this is a minority view among the meetings he has attended.
A practitioner reading of the November 2025 report as a saturation failure in a protective subsystem, plus a fair criticism: the writeup cannot tell you what the engineer was thinking, so it cannot support the local-rationality question.
The canonical statement of the action-item position, including the requirement that a user-affecting outage produce a tracked high-priority bug, and a warning that without formal tracking action items are forgotten and outages follow.
The sharpest published critique of error budgets as a learning mechanism: they price the risk you chose to take and say nothing about the risk nobody knew was there, which is the category every incident in section 4 belongs to.
The no-correlation finding between duration and severity, the skew argument against means, Allspaw's definition of shallow data, and the recommended replacements: service level objectives, coordination cost, and near-miss analysis.
Corpus size and shape, the 53% resolved within two hours figure, and the observation that near-miss reports are close to absent because nothing triggers a process and nobody wants to publish them.
198 sampled production failures across five systems. The 92% and 58% figures, the three trivial handler patterns, and a static checker that found 143 confirmed bugs in systems already using FindBugs and fault injection.
597 outages across 32 services over seven years, with the root-cause distribution and, more usefully, the honest accounting of how much of the public record says nothing. The recovery-chain argument is the paper's real contribution.
J. Paul Reed's three-month field study of how one high-performing organisation used post-incident artefacts, set against survey data showing 91% of practitioners define the meeting by its remediation list.
A consortium of Etsy, IBM, IEX and Ohio State analysing each other's real incidents. Introduces dark debt and the above-the-line model, and observes that postmortems reliably surface model surprises while staying narrow about everything except code.
Nash at SREcon22 Americas, 14 March 2022, presenting from a corpus of nearly 2,000 reports across 660 organisations. Slides and video are linked from the USENIX programme page.
A published transcript, which makes it citable line by line. Contains Lorin's Law on remediation items becoming latent conditions, Rosenthal on why start and end timestamps do not help, and the missing-denominator argument.
InfoQ's report of the QCon London talk, 23 April 2019, with the working definition of resilience as the capacity to handle the unanticipated, which is the case a catalogue of anticipated fixes cannot cover.
Amazon's published description of its Correction of Error mechanism: summary, impact, timeline, metrics, incident questions, 5 Whys, and action items, framed explicitly around avoiding recurrence.
The operational detail behind the concept page: iterate with the team first, then route to leadership for external Correction of Error documents, with review level scaled to company size.
The corpus itself, searchable, which is the only public place to compare incident reports across organisations. Used here to confirm the absence of published action-item and repeat-incident data.
Six rungs. The line between exercise and production work is between rungs three and four, where the artefact stops being a document and starts being a gate.
Take the Cloudflare 2019 and 2025 reports, the Google Cloud June 2025 report and the CrowdStrike RCA. Tabulate every committed action against the property it protects, not the component it names.
Done when: you can state, in one sentence, the property all four share and name at least three artefacts in your own system that have it. Teaches: that remediation lists are written against components, which is why they do not generalise.
List every path by which behaviour reaches production without a code deploy. Feature flags, config stores, model artefacts, rule sets, DNS, index definitions, entitlement data, pricing tables. For each, record whether it is staged, whether it can be reverted without a deploy, and whether the consumer validates it.
Done when: the table exists and at least one row fails all three columns. Teaches: that your release safety is scoped to code, and how much of production is not code.
Follow Etsy's structure: reconstruct the timeline first, park every improvement idea in a visible list, and do not commit to anything until the timeline is complete. Compare what the list contains against what you would have written in the first ten minutes.
Done when: at least one item on the parked list would not have been proposed under the old format. Teaches: that early commitment to a fix ends the investigation.
Query your issue tracker for post-incident items: opened, closed, closed as will-not-do, still open, bucketed by age. Publish it monthly. Nobody else has this number, which is why yours is worth having.
Done when: the age distribution is on a dashboard and the oldest open item has an explicit decision against it. Teaches: whether your process produces change or produces backlog.
Pick the lesson your organisation has learned twice. Encode it where changes are made: a required field in the change template, a lint rule, a CI check, a contribution-guide rule with tooling, in the shape of Envoy's runtime guards or the Kubernetes production readiness questionnaire.
Done when: a change that violates the rule fails automatically, and somebody who never read the incident report hits it. Teaches: the difference between a lesson stored in people and a lesson stored in the system.
Find an event that was caught before customer impact. Run the full review. Then check whether your severity criteria would ever have triggered it, and change them if not.
Done when: a near miss has produced a tracked change and the trigger criteria have been amended. Teaches: that the corpus everyone learns from is selected on damage, which is the worst possible sampling rule.
The queries that found the material above. The vocabulary matters more than the operators: postmortem writing has a house style, and these phrases are in it.
"we have already disabled" postmortem automation worldwidesite:aws.amazon.com/message/ summary of the service disruption"worst outage since" site:blog.cloudflare.com"latent race condition" OR "latent defect" incident report 2025"staged rollout" postmortem commitment site:blog.<vendor>.com"in the same manner used for other software""was not feature flag protected" OR "no feature flag" incident report"replicated globally within seconds" OR "propagated globally"site:handbook.gitlab.com incident review bar raiser corrective actions"correction of error" COE template 5 whys leadership review"production readiness review" path:keps rollback disabled"runtime guard" OR "reloadable_features" CONTRIBUTING behavioural change"action items" incident review "minority view" OR "I don't like""shallow data" incident MTTR Allspaw OR Nash"near miss" incident analysis software why so few reportsrepo:kubernetes/enhancements is:pr is:closed is:unmerged production readinessTwo mechanical notes from this hunt. First, the
GitHub REST search endpoint answers is:pr is:closed is:unmerged queries as JSON
when the web interface will not render, which is how the closed Kubernetes pull request in
section 4 was found. Second, several of the best sources answer a plain fetch with a 403 and
a normal browser user agent with a 200; USENIX, ACM Queue and GitLab's blog all behave this
way, and a research pass that gives up on the first 403 loses the primary sources and keeps
the summaries.