Evidence ledger
34 sources
Checked 17 Oct 2026
Evidence ledger
One row per claim in The delete lands one level up, on the path nobody guarded: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Tiers follow the implementation-archaeology grading
(postmortem > source/adr > casestudy > paper/talk > blog > vendor).
Access note (2026-10-17). This session ran in a cloud environment whose network policy
allows gitlab.com, raw.githubusercontent.com, github.com (git protocol only) and
cloud.google.com, and denies most other hosts. Sources on those hosts were fetched directly
and their quotes are copied from the fetched text. Rows marked [search] sit on denied hosts
(atlassian.com, aws.amazon.com, blog.cloudflare.com, engineering.fb.com, usenix.org, infoq.com,
brandur.org, about.gitlab.com, theregister.com, docs.aws.amazon.com, code.djangoproject.com and
others); their content was retrieved through the session's server-side search tool, quotes are
limited to the fragments retrieval returned verbatim, and anything beyond those fragments is
written in the guide as attributed paraphrase. Every link below was returned by retrieval or
fetched in this session.
Primary not retrievable. Microsoft's own Azure DevOps postmortem for 24 May 2023 (Eric Mattingly, devblogs.microsoft.com) was not returned by retrieval. Rows 9 to 11 cite the press accounts that quote it, and the guide says so on the card.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Atlassian | Post-Incident Review on the Atlassian April 2022 outage [search] | postmortem | 2022-04-29 | 2026-10-17 | https://www.atlassian.com/blog/atlassian-engineering/post-incident-review-april-2022-outage | Scope and window of the deletion | "883 sites (representing 775 customers)"; script "started running on April 5th at 07:38 UTC and was completed at 08:01 UTC"; outage "spanned up to 14 days for a subset of these customers" |
| 2 | Atlassian | Same PIR [search] | postmortem | 2022-04-29 | 2026-10-17 | (as #1) | Wrong identifiers: site IDs instead of app IDs; one script carried both modes | "Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site"; the script provided both the "mark for deletion" capability "used in normal day-to-day operations (where recoverability is desirable), and the 'permanently delete' capability"; executed "with the wrong execution mode and the wrong list of IDs" |
| 3 | Atlassian | Same PIR [search] | postmortem | 2022-04-29 | 2026-10-17 | (as #1) | Restore capability was built for a different granularity than the delete | lacked "the ability to select a large set of customer sites and restore all of their inter-connected products from backups to a previous point in time"; capabilities "focused on infrastructure, data corruption, single service events, or single-site deletions" |
| 4 | Atlassian | Same PIR [search] | postmortem | 2022-04-29 | 2026-10-17 | (as #1) | Remediation: universal soft delete as a precondition of global deletion | customer data and metadata "that has not gone through a soft-delete process cannot be deleted globally"; deletions "should be blocked or protected by layers of safeguards, including staged rollout and tested rollback plan for 'soft deletes'" |
| 5 | Atlassian | April 2022 outage update [search] | blog | 2022-04-12 | 2026-10-17 | https://www.atlassian.com/engineering/april-2022-outage-update | Restore throughput during the incident | restoring "in batches of up to 60 tenants at a time"; "end-to-end, it takes between 4 and 5 elapsed days to hand a site back to a customer" |
| 6 | Atlassian | August update on April 2022 outage PIR progress [search] | blog | 2022-08 | 2026-10-17 | https://www.atlassian.com/blog/announcements/pir-update-august2022 | Follow-through: DR exercise for site deletion scheduled | first DR site-deletion exercise planned for "the last quarter of 2022" |
| 7 | The Pragmatic Engineer | Inside the Longest Atlassian Outage of All Time [search] | blog | 2022-04 | 2026-10-17 | https://newsletter.pragmaticengineer.com/p/scoop-atlassian | Dissent on transparency of the PIR | "Atlassian published their postmortem, but refused to share the number of impacted users" |
| 8 | AWS | Summary of the Amazon S3 Service Disruption in US-EAST-1 [search] | postmortem | 2017-03-02 | 2026-10-17 | https://aws.amazon.com/message/41926/ | Capacity removal hit a larger set than intended; fix was rate and floor | command "intended to remove a small number of servers ... one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended"; the tool "allowed too much capacity to be removed too quickly"; tool changed to remove capacity "more slowly" with safeguards against going "below its minimum required capacity level"; "auditing our other operational tools to ensure we have similar safety checks" |
| 9 | The Register (reporting Microsoft's postmortem) | Azure DevOps outage in Brazil [search] | postmortem | 2023-06-03 | 2026-10-17 | https://www.theregister.com/2023/06/03/microsoft_azure_outage_brazil/ | The code change was a package migration | change replaced deprecated "Microsoft.Azure.Management" packages with "Azure.ResourceManager" NuGet packages |
| 10 | DatacenterDynamics (reporting Microsoft's postmortem) | Microsoft Azure outage in Brazil caused by typo [search] | postmortem | 2023-06 | 2026-10-17 | https://www.datacenterdynamics.com/en/news/microsoft-azure-outage-in-brazil-caused-by-typo/ | Database-delete call became server-delete; 17 databases; 10h21m; no data loss | snapshot deletion job "swapped out a call to delete the Azure SQL Database to one that deletes the Azure SQL Server"; "deleted all seventeen production databases for the scale unit"; offline "from 12:10 to 22:31 UTC"; aware "within 20 minutes"; "no data loss" |
| 11 | Techzine (reporting Microsoft's postmortem) | Typo caused hours-long Microsoft Azure DevOps outage [search] | postmortem | 2023-06 | 2026-10-17 | https://www.techzine.eu/news/devops/106909/typo-caused-hours-long-microsoft-azure-devops-outage | Ring 0 passed because no snapshots existed; restore needed Azure engineers; backup configs differed | "Sprint 222 was deployed internally (Ring 0) without incident due to the absence of any snapshot databases"; customers "cannot restore Azure SQL servers themselves"; deleted databases had differing backup configurations |
| 12 | Google Cloud | Sharing details on a recent incident impacting one of our customers | postmortem | 2024-05-24 | 2026-10-17 | https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident | Blank parameter defaulted to a one-year term with automatic deletion | "one input parameter was left blank when using an internal tool"; "the system assigned a then unknown default fixed 1 year term value"; "After the end of the system-assigned 1 year period, the customer's GCVE Private Cloud was deleted" |
| 13 | Google Cloud | Same | postmortem | 2024-05-24 | 2026-10-17 | (as #12) | The notification safeguard existed only on the customer path | "No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due a customer deletion request. Any customer-initiated deletion would have been preceded by a notification to the customer." |
| 14 | Google Cloud | Same | postmortem | 2024-05-24 | 2026-10-17 | (as #12) | Recovery came from copies outside the deleted object; remediation deleted the exception path | backups "stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental"; "We deprecated the internal tool that triggered this sequence of events"; safeguards described as "soft delete, advance notification, and human-in-the-loop" |
| 15 | GitLab | Postmortem of database outage of January 31 (source markdown) | postmortem | 2017-02-10 | 2026-10-17 | https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/30d10754fb46472e/source/posts/2017-02-10-postmortem-of-database-outage-of-january-31.html.md | Wipe ran on primary instead of secondary; 300 GB | "an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead"; "around 300 GB of data had already been removed" |
| 16 | GitLab | Same | postmortem | 2017-02-10 | 2026-10-17 | (as #15) | GitLab explicitly declined command guardrails in favour of recoverability | "Our main focus is to improve disaster recovery, and making it more obvious as to what host you're using; instead of preventing production engineers from running certain commands"; aliasing rm "would only protect themselves against accidentally running rm -rf /important-data, not against disk corruption" |
| 17 | GitLab | Issue #1094: Update PS1 across all hosts | source | 2017-02-01 | 2026-10-17 | https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/1094 | The prompt hid the environment and truncated the host | "it doesn't show the environment one is in (staging vs production), the colours are the same on staging/production, and the hostname is truncated to the first segment"; proposal "Use bold red for production" |
| 18 | Cloudflare | Cloudflare incident on February 6, 2025 [search] | postmortem | 2025-02-06 | 2026-10-17 | https://blog.cloudflare.com/cloudflare-incident-on-february-6-2025/ | Abuse remediation on one bucket disabled the whole R2 Gateway | "human error and insufficient validation safeguards during a routine abuse remediation for a report about a phishing site hosted on R2"; action "led to disabling the production R2 Gateway service responsible for the R2 API"; 59 minutes; no data loss or corruption |
| 19 | Hackmag (reporting Cloudflare's remediation) | Failed attempt to block phishing link results in massive Cloudflare outage [search] | blog | 2025-02 | 2026-10-17 | https://hackmag.com/news/cloudflare-r2-out | Remediation removed the capability from the role | the option to disable systems "removed from the interface used by abuse remediation staff"; restrictions "added to the Admin API to prevent internal accounts from disabling services" |
| 20 | Spotify | How Spotify Accidentally Deleted All Its Kube Clusters with No User Impact (KubeCon EU 2019 keynote slides) | talk | 2019-05-22 | 2026-10-17 | https://raw.githubusercontent.com/sbueringer/kubecon-slides/master/slides/2019-kubecon-eu/Keynote%20How%20Spotify%20Accidentally%20Deleted%20All%20its%20Kube%20Clusters%20with%20No%20User%20Impact%20-%20David%20Xia,%20Infrastructure%20Engineer,%20Spotify%20-%20KubeCon%20Europe%202019%20Keynote%20-%20David%20Xia%20-%20How%20Spotify%20Accidentally%20Deleted%20All%20Its%20Kube%20Clusters%20with%20No%20User%20Impac.pdf | First deletion and restore time | slide 11: "I deleted a 50-node production cluster running dozens of workloads"; slide 13: restoration "took 3.25 hours", "cluster creation process wasn't resumable, all or nothing" |
| 21 | Spotify | Same slides | talk | 2019-05-22 | 2026-10-17 | (as #20) | The remediation for deletion caused the second deletion | slides 14-15: "trying to prevent accidental cluster deletions by codifying them"; "unknowingly modified global state during review builds"; "two PRs merged out of order" |
| 22 | Spotify | Same slides | talk | 2019-05-22 | 2026-10-17 | (as #20) | No user impact because of partial migration and service discovery on pod IPs | "we recommended teams only migrate some but not all of each service's instances to K8s"; "clients only got a list of non-K8s instances"; "we had already tested restoring from these" |
| 23 | Netflix | Architecting a Centralized Platform for Data Deletion at Netflix (QCon SF 2025) [search] | talk | 2025-11 | 2026-10-17 | https://infoq.com/presentations/architecting-deletion-system | Deletion as a platform with audit loops and tombstone control | talk covers "controlling tombstone accumulation, building continuous audit loops, and gaining trust with a centralized platform"; runs 49:41 |
| 24 | InfoQ (reporting Netflix talk) | Netflix data deletion [search] | blog | 2025-11 | 2026-10-17 | https://infoq.com/news/2025/11/netflix-data-deletion/ | Scale figure; resurrection named as a failure class | "76.8 billion row deletions across 1,300 datasets with zero data loss incidents"; deleted data "can reappear due to misconfiguration, extended node downtime, or synchronization issues"; deletes treated as "low-priority requests" |
| 25 | Zero Touch Prod: Towards Safer and More Secure Production Environments (SREcon19 EMEA slides) [search] | talk | 2019-10 | 2026-10-17 | https://www.usenix.net/sites/default/files/conference/protected-files/srecon19emea_slides_wolafka.pdf | Ad-hoc human changes named as an outage class; every change automated, validated, or break-glass | outage scenario titled "I never make tipos"; "No review/approval or proper change control"; every production change automated, "pre-validated by software", or through an audited break-glass mechanism | |
| 26 | Meta / USENIX | DelF: Safeguarding deletion correctness in Online Social Networks (USENIX Security 2020) [search] | paper | 2020-08 | 2026-10-17 | https://www.usenix.org/conference/usenixsecurity20/presentation/cohn-gordon | Restoration logs undo wrong top-level deletes and mis-annotated edges | "Every deletion in DELF generates a write-ahead log referred to as its restoration log"; used "to recover from application bugs that trigger the deletion of the wrong top-level object and from mistakes in edge type annotations that declare deep or refcount rather than shallow" |
| 27 | Meta / USENIX | Same paper [search] | paper | 2020-08 | 2026-10-17 | (as #26) | Mis-annotation rate; over-deletion twice as common as over-retention; tens of recoveries | developers "incorrectly annotate approximately 2 edge types every day"; mistakes "that result in inadvertent deletion are approximately twice as common than those that result in inadvertent data retention"; "enabling timely recovery in tens of incidents where user data was inadvertently deleted" |
| 28 | Meta | DELF: Safeguarding deletion correctness in online social networks (engineering blog) [search] | blog | 2020-08-12 | 2026-10-17 | https://engineering.fb.com/2020/08/12/security/delf/ | Scale and undo window | "supports the execution of billions of deletions every day"; restoration log "enables engineers to perform an 'undo' operation for a limited time period" |
| 29 | Gunawi et al. | Why Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages (SoCC 2016) [search] | paper | 2016-10 | 2026-10-17 | https://www.ece.iastate.edu/%7Emai/docs/papers/2016_SoCC_COS.pdf | The broad outage corpus exists and covers root causes and fix procedures | "1247 headline news and public post-mortem reports that detail 597 unplanned outages that occurred within a 7-year span from 2009 to 2015" |
| 30 | GitLab | Group and project deletions (architecture design document) | adr | 2025-04-04 | 2026-10-17 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/group_and_project_deletions/_index.md | Motivation and the unified pending-deletion state | "The current system presents significant challenges, including accidental permanent deletions, high support burden for recovery requests, and locked namespace paths"; "Projects in user namespaces are deleted permanently without any recovery period" |
| 31 | GitLab | Same design document | adr | 2025-04-04 | 2026-10-17 | (as #30) | Rejected alternative and its stated cost | Alternative "Permanent Deletion Restriction": only admins permanently delete early; cons include "Extending the pending deletion period from 7 to 30 days increases storage requirements approximately fourfold" and "Users unable to immediately and permanently remove sensitive data without admin involvement" |
| 32 | GitLab | Same design document | adr | 2025-04-04 | 2026-10-17 | (as #30) | Children follow the parent; path freed by renaming | "the item and all its child items, if any, move to 'pending deletion' state"; "System automatically renames the namespace to free up the original path"; "Items pending deletion still count towards usage quotas" |
| 33 | GitLab | Epic 17208: Down-tier delayed deletion to core | source | 2025-03-21 | 2026-10-17 | https://gitlab.com/groups/gitlab-org/-/epics/17208 | The protection had been a paid-tier feature; goal is all tiers | "Extend the pending deletion state to all pricing tiers, including free tier projects and groups"; "Reduced support ticket volume for deletion recovery requests"; closed 2025-05-07 |
| 34 | GitLab | Epic 17375: Extend pending deletion period from 7 to 30 days on SaaS | source | 2025-04-04 | 2026-10-17 | https://gitlab.com/groups/gitlab-org/-/epics/17375 | Window extended; storage analysis a requirement | "Increase the pending deletion period from 7 to 30 days for all entities on GitLab SaaS"; "Storage impact analysis to accommodate increased retention"; closed 2025-07-10 |
| 35 | GitLab | Delayed deletion (blog) [search] | blog | 2023-04-03 | 2026-10-17 | https://about.gitlab.com/blog/delayed-deletion/ | Delayed deletion became default in 16.0 for paid tiers because support kept recovering deletions | delayed deletion default "in GitLab 16.0 for Ultimate and Premium customers to prevent accidental deletions"; support "often helps customers recover accidentally deleted groups or projects" |
| 36 | GitLab | Improving GitLab deletion flow (blog) [search] | blog | 2025-07-14 | 2026-10-17 | https://about.gitlab.com/blog/improving-gitlab-deletion-flow-what-to-expect-in-coming-months/ | 30 days live on GitLab.com; admin deletions still to follow | "On July 10, 2025, GitLab increased the pending deletion period from 7 to 30 days on GitLab.com"; "Admin-initiated deletions will follow the same pending-deletion process" listed as coming next |
| 37 | Kubernetes | KEP-3895: Interactive(-i) flag to kubectl delete | adr | 2023-03-01 | 2026-10-17 | https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-cli/3895-kubectl-delete-interactivity/README.md | Default confirmation ruled out by compatibility; opt-in flag shipped stable in v1.30 | "Due to the backwards compatibility concerns, seeking confirmation from a user before proceeding to genuinely delete as default, is out of the table"; kep.yaml: status implemented, stage stable, latest-milestone v1.30 |
| 38 | Kubernetes | KEP-3895 alternatives section | adr | 2023-03-01 | 2026-10-17 | (as #37) | Rejected design: per-resource yes-to-all | "Yes to all means that 'you can delete the rest, even though I don't know'" |
| 39 | Kubernetes | [RFC] Protecting users of kubectl delete (kubernetes-dev) [search] | source | 2023 | 2026-10-17 | https://groups.google.com/g/kubernetes-dev/c/y4Q20V3dyOk/m/vNbttyY-BgAJ | Long-requested default confirmation repeatedly declined | "Confirmation when deleting with --all and --all-namespaces is a long requested feature but we've historically determined this to be a breaking change and declined to implement" |
| 40 | Kubernetes | KEP-5080: Ordered Namespace Deletion | adr | 2024-01-27 | 2026-10-17 | https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-api-machinery/5080-ordered-namespace-deletion/README.md | Order of a cascading delete is itself a safety property | "The current deletion process is semi-random, which may lead to security gaps or unintended behavior, such as Pods persisting after the deletion of their associated NetworkPolicies"; status implemented, stable, v1.34 |
| 41 | discard (Ruby) | README: Why not paranoia or acts_as_paranoid? | source | 2026 (README at master) | 2026-10-17 | https://raw.githubusercontent.com/jhawthorn/discard/master/README.md | A former paranoia maintainer rejects default-scope soft delete | "I've worked with and have helped maintain paranoia for a while. I'm convinced it does the wrong thing for most cases"; "dependent: :destroy associations are deleted when performing soft-destroys" |
| 42 | Django | Ticket #16567: Feature Req: Soft Delete (wontfix) [search] | source | 2011 | 2026-10-17 | https://code.djangoproject.com/ticket/16567 | Framework declined to own soft delete | ticket closed as wontfix; reporter listed the overrides needed: model delete(), queryset delete(), filtering manager |
| 43 | Django | Ticket #33375 (wontfix) [search] | source | 2022 | 2026-10-17 | https://code.djangoproject.com/ticket/33375 | Default-manager soft delete breaks the restore path in admin | "The id field in the changelist_formset uses the _default_manager" so soft-deleted rows could not be restored through list-editable admin |
| 44 | brandur.org | Soft deletion probably isn't worth it [search] | blog | 2022-07 | 2026-10-17 | https://brandur.org/soft-deletion | Row-level soft delete is rarely used to undelete and costs foreign keys | "never once, in ten plus years, did anyone at any of these places ever actually use soft deletion to undelete something"; "foreign keys are effectively lost"; proposes a deleted_record table with a JSON dump |
| 45 | brandur.org | Deleted record insert (fragment) [search] | blog | 2024 | 2026-10-17 | https://brandur.org/fragments/deleted-record-insert | The author migrated off deleted_at | alternative "doesn't interfere with mainline code (no need to include a deleted_at IS NULL predicate in every live query, no foreign key problems)" |
| 46 | SRE book ch. 26, Data Integrity: What You Read Is What You Wrote [search] | blog | 2016 | 2026-10-17 | https://sre.google/sre-book/data-integrity/ | Soft deletion as first layer before backups and validation | soft deletion ("lazy deletion" for developer APIs) as the first layer, backups and recovery second, data validation third | |
| 47 | HashiCorp | Terraform lifecycle meta-argument (v1.1 docs) [search] | vendor | 2022 | 2026-10-17 | https://developer.hashicorp.com/terraform/language/v1.1.x/meta-arguments/lifecycle | prevent_destroy is removed with the block it protects | "this setting does not prevent the remote object from being destroyed if the resource block were removed from configuration entirely: in that case, the prevent_destroy setting is removed along with it" |
| 48 | AWS | ScheduleKeyDeletion (KMS developer guide) [search] | vendor | current | 2026-10-17 | https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys-scheduling-key-deletion.html | Mandatory waiting period for an irreversible delete | waiting period default 30 days, value "between 7 and 30, inclusive"; key unusable in PendingDeletion; CancelKeyDeletion during the window |
| 49 | Google Cloud | Delete and restore projects [search] | vendor | current | 2026-10-17 | https://docs.cloud.google.com/resource-manager/docs/delete-restore-projects | The container window is weaker than it looks: some children go sooner | 30-day recovery period; "Some services—such as Cloud Storage and Pub/Sub—delete resources much sooner. These might not be fully recoverable even if the project is restored." |
| 50 | Microsoft (Azure China docs mirror) | Azure Key Vault soft-delete overview [search] | vendor | current | 2026-10-17 | https://docs.azure.cn/en-us/key-vault/general/soft-delete-overview | Retention fixed once set; purge protection | retention "7 to 90 days"; "Once the retention policy interval is set and saved, it can't be changed for that vault" |
| 51 | Trend Micro knowledge base | Key Vault purge protection enabled [search] | vendor | 2025 | 2026-10-17 | https://trendmicro.com/trendaivisiononecloudriskmanagement/knowledge-base/azure/KeyVault/purge-protection-enabled.html | Soft delete made non-optional | "as of February 2025, Microsoft enabled soft delete protection on all key vaults. Users can no longer opt out of or turn off soft delete" |