Evidence ledger
34 sources
Checked 28 Aug 2026
Evidence ledger
One row per claim in The rule was already written down: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: The rule was already written down — how engineering organisations convert a production failure into a durable design change.
Every row is one claim taken from one artefact. Every URL was fetched during research on 2026-08-28. The "supporting quote" column is copied, not paraphrased; where a figure is taken rather than a sentence, the figure is reproduced exactly as published.
Tiers, strongest first: postmortem, source, adr, casestudy, blog, paper, talk,
vendor.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Cloudflare | Cloudflare incident on November 18, 2025 | postmortem | 2025-11-18 | 2026-08-28 | https://blog.cloudflare.com/18-november-2025-outage/ | A ClickHouse permissions change doubled a Bot Management feature file, which exceeded a hardcoded 200-feature limit and panicked the Rust proxy; core traffic stopped for 3h46m. | "Today was Cloudflare's worst outage since 2019… in the last 6+ years we've not had another outage that has caused the majority of core traffic to stop flowing through our network." |
| 2 | Cloudflare | Cloudflare incident on November 18, 2025 | postmortem | 2025-11-18 | 2026-08-28 | https://blog.cloudflare.com/18-november-2025-outage/ | The team held a wrong hypothesis for over two hours: impact began 11:20 UTC, the correct cause was identified at 13:37 UTC. | Initial belief: a "hyper-scale DDoS attack"; the status-page outage led teams to "believe that an attacker may be targeting both our systems". Correct identification 13:37 UTC. |
| 3 | Cloudflare | Cloudflare incident on November 18, 2025 | postmortem | 2025-11-18 | 2026-08-28 | https://blog.cloudflare.com/18-november-2025-outage/ | The stated remediation is to treat internal configuration as untrusted input and add global kill switches. | "Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input"; "Enabling more global kill switches for features". |
| 4 | Cloudflare | Details of the Cloudflare outage on July 2, 2019 | postmortem | 2019-07-12 | 2026-08-28 | https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ | Six years before the 2025 outage, Cloudflare committed to staged rollout of globally propagated rule content. | "Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks." |
| 5 | Cloudflare | Details of the Cloudflare outage on July 2, 2019 | postmortem | 2019-07-12 | 2026-08-28 | https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ | The 2019 outage was 27 minutes and cost roughly 80% of traffic; a regex with excessive backtracking exhausted CPU globally. | WAF deployed 13:42 UTC, terminated globally 14:07 UTC, re-enabled 14:52 UTC; "80% of traffic lost". |
| 6 | Amazon Web Services | Summary of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) | postmortem | 2025-10-23 | 2026-08-28 | https://aws.amazon.com/message/101925/ | A latent race condition between two DNS Enactor instances deleted every IP address for the DynamoDB regional endpoint; recovery needed manual operator intervention. | "The root cause of this issue was a latent race condition in the DynamoDB DNS management system"; "This situation ultimately required manual operator intervention to correct." |
| 7 | Amazon Web Services | Summary of the Amazon DynamoDB Service Disruption | postmortem | 2025-10-23 | 2026-08-28 | https://aws.amazon.com/message/101925/ | The first remediation was to switch the automation off worldwide, before fixing it. | "We have already disabled the DynamoDB DNS Planner and the DNS Enactor automation worldwide. In advance of re-enabling this automation, we will fix the race condition scenario and add additional protections to prevent the application of incorrect DNS plans." |
| 8 | Google Cloud | Google Cloud Service Health incident report, 12 June 2025 (Service Control) | postmortem | 2025-06-13 | 2026-08-28 | https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW | A new quota-policy code path shipped without a feature flag and without error handling; a policy row with blank fields crashed Service Control binaries globally. | The feature was "not… feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash." Policy data "was replicated globally within seconds". |
| 9 | Google Cloud | Google Cloud incident report, 12 June 2025 | postmortem | 2025-06-13 | 2026-08-28 | https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW | The remediation adopts, in 2025, the rule open-source projects codified in 2019: every change to a critical binary must be flag-guarded and default-off. | "Enforce all changes to critical binaries to be feature flag protected and disabled by default"; "Modularize Service Control's architecture, so the functionality is isolated and fails open". |
| 10 | CrowdStrike | External Technical Root Cause Analysis — Channel File 291 | postmortem | 2024-08-06 | 2026-08-28 | https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf | A 21-versus-20 parameter mismatch in content that bypassed the sensor's staged release process triggered a latent out-of-bounds read on every machine that received it. | Mitigation list includes "Validate the number of input fields in the IPC Template Type at sensor compile time" and "Implement staged deployment for Template Instances". |
| 11 | GitLab | Postmortem of database outage of January 31 | postmortem | 2017-02-10 | 2026-08-28 | https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/ | Every one of five backup and replication mechanisms failed, and the failure of the primary one was silent because a version mismatch aborted pg_dump and nobody received the alert mail. |
"When we went to look for the pg_dump backups we found out they were not there. The S3 bucket was empty"; roughly 300 GB removed; "roughly 5,000 projects, 5,000 comments and 700 new user accounts" lost. |
| 12 | GitLab | Postmortem of database outage of January 31 | postmortem | 2017-02-10 | 2026-08-28 | https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/ | GitLab structured the public analysis around The 5 Whys, splitting it into two problems: the outage and the eighteen-hour recovery. | "To analyse the root cause of these problems we'll use a technique called 'The 5 Whys'… Problem 1: GitLab.com was down for about 18 hours." |
| 13 | Envoy (CNCF) | CONTRIBUTING.md, runtime guarded features | source | 2019-03-13 (introduced) | 2026-08-28 | https://github.com/envoyproxy/envoy/blob/main/CONTRIBUTING.md | Envoy requires high-risk and user-visible behavioural changes to ship behind a runtime flag that is safe to flip on a running process. | Guards use the form Runtime::runtimeFeatureEnabled("envoy.reloadable_features.my_feature_name"); flags with that prefix must remain "safe to flip true or false on running Envoy instances". |
| 14 | Envoy (CNCF) | PR #6134 · runtime: codifying runtime guarded features | source | 2019-03-13 | 2026-08-28 | https://github.com/envoyproxy/envoy/pull/6134 | The runtime-guard rule was written into the contributor contract by a maintainer, as documentation plus tooling, not left to reviewer judgement. | Author alyssawilk: "Documenting best practices and adding some tooling for guarding high risk features." |
| 15 | Envoy (CNCF) | Issue #20212 · reject runtime variables with the runtime guard naming if they're not known flags | source | 2022-03-03, open | 2026-08-28 | https://github.com/envoyproxy/envoy/issues/20212 | The safety control has its own silent-failure mode: an operator's override of a removed guard becomes a no-op, and the issue proposing hard rejection has been open for over four years. | "if folks override the default behavior they (previously) have no warning when the code paths are removed and their override becomes a no-op… I propose we outright reject configs with unknown flags, in case folks aren't paying attention to their logs." |
| 16 | Etsy | Debriefing Facilitation Guide, section 4 | source | 2016 | 2026-08-28 | https://github.com/etsy/DebriefingFacilitationGuide/blob/master/guide/04-structure-and-preparation.md | Etsy's open-sourced facilitation guide explicitly de-prioritises remediation items and denies that prevention is the goal. | "Preventing the future is not the goal—learning is." And: "Having remediation items is not a requirement for a good debriefing." |
| 17 | Kubernetes | PR #5645 · KEP-3926: updating the PRR questionnaire (closed, unmerged) | source | opened 2025-10-09, closed 2026-01-29 | 2026-08-28 | https://github.com/kubernetes/enhancements/pull/5645 | A change to the production-readiness gate stalled 97 days waiting on the same scarce approver pool the gate itself depends on, went stale, and was closed as a duplicate. | k8s-triage-robot applied the stale label on 2026-01-14; soltysh closed it: "It seems all the changes to PRR were copied over to #5739 so I'm going to close this as a duplicate." |
| 18 | Kubernetes | KEP-1194: Production Readiness Review process | adr | 2019-07-31 | 2026-08-28 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/1194-prod-readiness | Kubernetes turned a class of recurring operational oversight into a mandatory review gate rather than a recommendation. | "The review process is intended to ensure that features merging into Kubernetes are observable and supportable, can be safely operated in production environments, and can be disabled or rolled back in the event they cause increased failures in production." |
| 19 | Kubernetes | KEP template · Production Readiness Review questionnaire | adr | ongoing | 2026-08-28 | https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/README.md | The gate is a fixed questionnaire whose first section is Feature Enablement and Rollback, and it is checked mechanically against kep.yaml. |
Questions include "Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?", "How can a rollout or rollback fail? Can it impact already running workloads?", "What specific metrics should inform a rollback?" |
| 20 | GitLab | Handbook · Incident Review | adr | current (checked 2026-08-28) | 2026-08-28 | https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/ | GitLab gates review closure on an independent Bar Raiser and asks explicitly whether the lesson generalises beyond the incident that produced it. | "Are there areas outside of this specific impact that need to be addressed in the same fashion?"; "A review must have Bar Raiser approval before it can be closed." |
| 21 | GitLab | Handbook · Incident Review, timeline | adr | current (checked 2026-08-28) | 2026-08-28 | https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/ | GitLab publishes an explicit clock on the analysis, not on the corrective actions. | "It is expected that the incident review will be closed within 5 working days of the incident resolution. Customers should expect the public RCA to be available within 7 days of the incident resolution for Severity 1 incidents." |
| 22 | Uptime Institute | Annual Outage Analysis 2025 (7th edition) | casestudy | 2025-05-06 | 2026-08-28 | https://uptimeinstitute.com/about-ui/press-releases/uptime-announces-annual-outage-analysis-report-2025 | The dominant human-error failure mode is procedures being ignored or being wrong, which is precisely the remediation category postmortems most often produce. | "Nearly 40% of organizations have suffered a major outage caused by human error over the past three years. Of these incidents, 85% stem from staff failing to follow procedures or from flaws in the processes and procedures themselves." |
| 23 | Adaptive Capacity Labs | Some Observations On the Messy Realities of Incident Reviews (Richard I. Cook) | blog | 2019-06-17 | 2026-08-28 | https://www.adaptivecapacitylabs.com/2019/06/17/some-observations-on-the-messy-realities-of-incident-reviews/ | Counting action items is a statement about what management wants from the review, not about what the review found. | "A manager who complains that too few action items were produced has revealed his/her real interest: the reduction of an incident to a manageable discrete list of things to be done." |
| 24 | Adaptive Capacity Labs | How Learning is Different Than Fixing (John Allspaw) | blog | 2020-05-06 | 2026-08-28 | https://www.adaptivecapacitylabs.com/2020/05/06/how-learning-is-different-than-fixing/ | The higher return comes from understanding the event, not from enumerating fixes for the parts involved in it. | "Rather than focusing on simply identifying fixes for parts involved in the event and instead focusing on developing a richer understanding of the event, a much greater ROI the effort will result." |
| 25 | Surfing Complexity (Lorin Hochstein) | Why I don't like discussing action items during incident reviews | blog | 2024-09-28 | 2026-08-28 | https://surfingcomplexity.blog/2024/09/28/why-i-dont-like-discussing-action-items-during-incident-reviews/ | The review meeting is the only scarce resource; action items get planning time anyway, mental models do not. | "updating people's mental models will have a more significant positive impact on the system than discussing action items"; "Judging from the incident review meetings I've attended throughout my career, this is a minority view." |
| 26 | Surfing Complexity (Lorin Hochstein) | Brief thoughts on the recent Cloudflare outage | blog | 2025-11-26 | 2026-08-28 | https://surfingcomplexity.blog/2025/11/26/brief-thoughts-on-the-recent-cloudflare-outage/ | The November 2025 outage is a saturation failure in a protective subsystem, and the postmortem cannot tell you what the engineer was thinking. | "A subsystem designed to protect surprisingly inflicts harm"; "There's nothing in the writeup that tells us what the programmer was thinking." |
| 27 | SRE Workbook, Postmortem Culture: Learning from Failure | blog | 2018 | 2026-08-28 | https://sre.google/workbook/postmortem-culture/ | Google's stated bar is that a postmortem without action is worthless to users, and that user-affecting outages must produce a tracked high-priority bug. | "To our users, a postmortem without subsequent action is indistinguishable from no postmortem"; "all postmortems which follow a user-affecting outage must have at least one P[01] bug associated with them". | |
| 28 | Will Gallego | Resilience Engineering and Error Budgets | blog | 2019-02-23 | 2026-08-28 | https://willgallego.com/2019/02/23/resilience-engineering-and-error-budgets/ | An error budget prices risk you chose to take and is silent about risk nobody knew was there. | "Critical failures can happen not because we've chosen to take a risk, but because we don't even know the risk is there to begin with." |
| 29 | InfoQ / Verica | Moving Past Simple Incident Metrics: Courtney Nash on the VOID | blog | 2023-02-14 | 2026-08-28 | https://www.infoq.com/articles/incident-metrics-void/ | Across the VOID corpus there is no correlation between how long an incident lasts and how bad it was, so duration-based metrics are not measuring reliability. | "There is no correlation detected between incident duration and incident severity"; Allspaw calls data that "underrepresents the uniqueness of incidents" shallow data. |
| 30 | InfoQ / Verica | Analyzing Incident Data across Organizations: Courtney Nash on the VOID | blog | 2022-06-28 | 2026-08-28 | https://www.infoq.com/articles/analyzing-incident-data/ | Near-miss reports are almost absent from the public corpus, which is the category with the most learning value per unit of damage. | "At the time of writing, the database contains over 1800 reports from close to 600 organizations"; "Over half (53%) of all reported incidents are externally resolved within two hours." |
| 31 | Ding Yuan et al., University of Toronto | Simple Testing Can Prevent Most Critical Failures (OSDI '14) | paper | 2014-10 | 2026-08-28 | https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf | Catastrophic failures are overwhelmingly failures of the code written to handle failure, and most were reachable by testing that needed no design knowledge. | "almost all (92%) of the catastrophic system failures are the result of incorrect handling of non-fatal errors explicitly signaled in software"; "in 58% of the catastrophic failures, the underlying faults could easily have been detected through simple testing of error handling code". |
| 32 | Haryadi S. Gunawi et al. | Why Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages (SoCC '16) | paper | 2016-10 | 2026-08-28 | https://dl.acm.org/doi/10.1145/2987550.2987583 | Most public outage records do not say what caused the outage or what was done about it, so the industry's shared corpus is far thinner than it looks. | "355 outages (out of the total 597) have UNKNOWN root causes"; "only 40% outage descriptions reveal root causes and only 24% reveal fix procedures". |
| 33 | Haryadi S. Gunawi et al. | Why Does the Cloud Stop Computing? (SoCC '16) | paper | 2016-10 | 2026-08-28 | https://dl.acm.org/doi/10.1145/2987550.2987583 | The hidden single point of failure is the recovery chain itself: detection, failover code and the backup component all have to be correct. | "the No-SPOF principle is not merely about redundancies, but also about the perfection of failure recovery chain: complete failure detection, flawless failover code, and working backup components." |
| 34 | J. Paul Reed | Beyond the Fix-it Treadmill (ACM Queue 17:6) | paper | 2019-12 | 2026-08-28 | https://queue.acm.org/detail.cfm?id=3380780 | Nine out of ten practitioners define the post-incident meeting by the remediation list it produces; the high-performing organisation studied did not. | "91 percent of respondents consider collection and recording of remediation items to be the core purpose of their post-incident analysis meetings and the artifacts produced from those meetings." |
| 35 | SNAFUcatchers (Etsy, IBM, IEX, Ohio State) | STELLA: Report from the SNAFUcatchers Workshop on Coping With Complexity | paper | 2017-03 | 2026-08-28 | https://snafucatchers.github.io/ | Postmortems reliably surface the fact that people's model of the system was wrong, and just as reliably stay narrow about everything except code. | "We see repeatedly that postmortems generate I-didn't-know-that-the-system-worked-that-way experiences"; postmortems are often "prosaic to the point of being narrow". |
| 36 | Courtney Nash (Verica) | Tales from the VOID: The Scary Truth about Incident Metrics, SREcon22 Americas | talk | 2022-03-14 | 2026-08-28 | https://www.usenix.org/conference/srecon22americas/presentation/nash | The talk argues from a 2,000-report corpus that MTTR is unreliable and that root-cause framing suppresses the reporting that would prevent the next incident. | "we demonstrate how unreliable MTTR can be, and how RCA can lead to environments where people are less likely to admit mistakes and speak up about things that could lead to future incidents." |
| 37 | Google SRE Prodcast | Human Factors in Complex Systems, with Casey Rosenthal and John Allspaw | talk | season 3 (transcript) | 2026-08-28 | https://sre.google/prodcast/transcripts/sre-prodcast-03-12/ | Remediation items generate their own latent conditions, and incident counts are meaningless without the denominator of comparable non-incidents. | Allspaw on "Lorin's Law": a "post-incident remediation item that turns into yet a new set of latent conditions". Rosenthal: "getting the CTO to invest more resources is immeasurably more impactful… And that's never going to come from an action item list." Allspaw: "the denominator is missing. Oh, you've had six incidents this week. Out of how many?" |
| 38 | InfoQ | Allspaw: Resilience Engineering, QCon London 2019 | talk | 2019-04-23 | 2026-08-28 | https://www.infoq.com/news/2019/04/allspaw-resilience-engineering/ | Resilience is about handling the situation nobody anticipated, which is why a catalogue of anticipated fixes is the wrong artefact to optimise. | "Resilience is aimed at setting and keeping conditions such that unforeseen, unanticipated, unexpected and fundamentally surprising situations can be handled." |
| 39 | Amazon Web Services | Correction of Error (COE) — AWS Well-Architected Framework | vendor | current (checked 2026-08-28) | 2026-08-28 | https://wa.aws.amazon.com/wat.concept.coe.en.html | Amazon's published process names corrective action, not understanding, as the object of the exercise, and routes serious COEs to leadership review. | The COE process is described as a mechanism for learning from an event "to avoid reoccurrences in the future", built around summary, impact, timeline, metrics, incident questions, 5 Whys, and action items. |
| 40 | Amazon Web Services | Creating a correction of errors document | vendor | 2022 (AWS Cloud Operations Blog) | 2026-08-28 | https://aws.amazon.com/blogs/mt/creating-a-correction-of-errors-document/ | The COE document is templated and reviewed up a management chain, which is what makes it durable and also what makes it optimise for defensibility. | "After completing the document, you should schedule an internal team review… For external COEs, leadership review follows, which can involve a senior manager, director, VP, or CTO." |
Coverage and gaps
What the record supports well. Mechanism. The large providers publish enough detail about individual incidents to reconstruct cause chains, detection delays and committed remediations with confidence. Rows 1 to 12 are all first-party, timestamped and specific.
What the record supports weakly. Process design. Rows 18 to 21 and 39 to 40 describe what organisations say their post-incident process is. No public source describes what any of these processes actually produced over a period of time.
What the record does not support at all. Outcome. No organisation has published:
- an action-item completion rate for post-incident corrective actions;
- a repeat-incident rate, defined consistently, tracked over time;
- a before-and-after comparison of incident rate against the introduction of a review process;
- the age distribution of open corrective actions.
Aggregator blogs and vendor marketing pages circulate figures such as "30% of action items are completed" and "repeat rate fell from 45% to 12%". None of them cites a primary source, and none was included here. The absence is itself the finding, and it is stated in the guide.
Addendum
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 41 | Kubernetes SIG Architecture | Production readiness review, process document | adr | current (checked 2026-08-28) | 2026-08-28 | https://github.com/kubernetes/community/blob/main/sig-architecture/production-readiness.md | The gate is deliberately staffed from outside the owning special interest group, and the dry runs showed that outsider view catches items the owners miss. | The review ensures features are "observable, scalable and supportable, can be safely operated in production environments, and can be disabled or rolled back in the event they cause increased failures in production"; reviews are "done by a separate team, apart from the SIG leads", and "experience through dry runs in 1.17-1.20 have shown that this slightly 'outsider' view helps identify otherwise missed items." |