Seven published deletions, from Atlassian, Microsoft, Google, Cloudflare, AWS, GitLab and Spotify, read side by side with the deletion designs of Meta, Netflix, GitLab and Kubernetes. None of the seven was a customer deleting their own data. Every one ran on the operator's own housekeeping path, and in four of them the delete hit the container of the thing it was aimed at. After reading, an architect can say where their system's undo window lives, whether it covers the path their own engineers use, and how long a restore takes compared with the delete that needs it.
Removing something on purpose, at scale, without removing the thing above it or beside it, and being able to put it back when you do.
This guide was picked because the evidence matured in the last eighteen months. GitLab published a design document in April 2025 that rebuilt its whole deletion flow and wrote down the alternative it rejected; it raised its undo window from seven to thirty days on 10 July 2025. Microsoft made Key Vault soft delete impossible to switch off in February 2025. Cloudflare disabled its own R2 storage gateway in February 2025 while trying to take down one phishing bucket, and Netflix described a deletion platform at QCon San Francisco in November 2025. Older incidents (AWS S3 in 2017, GitLab in 2017, Atlassian in 2022, Azure DevOps in 2023, Google Cloud in 2024) supply the failures. Read together, they say something none of them says alone.
The safeguards existed. They were attached to the wrong caller. Google's report says it in one sentence: no notification was sent because the deletion came from an operator's blank parameter, "not due a customer deletion request. Any customer-initiated deletion would have been preceded by a notification." Atlassian's script carried a recoverable "mark for deletion" mode for normal operations and a permanent mode beside it. Azure customers "cannot restore Azure SQL servers themselves". In every incident here, the undo machinery protected the customer's path and the outage came through the operator's.
The second pattern is about where the delete lands. In four of the seven incidents the operation hit the parent of its intended target: an app became its site at Atlassian, a database became its server at Microsoft, one bucket's endpoint became the whole R2 gateway at Cloudflare, and a few billing servers became part of the S3 index fleet at AWS. I am calling this scope inflation; none of the sources names it, and each describes it as a typo or a wrong list. It is not random. Identifiers for parent and child are usually the same shape, both are valid inputs to the same API, and the credential that may delete the child almost always may delete the parent.
Scope. This guide covers deliberate destructive operations in production: deleting tenants, resources, data and capacity, by people, scripts and scheduled jobs. It does not cover backup and restore drills (the earlier guide in this series, "A backup is a claim, a restore is the proof", does), privacy-driven erasure law, ransomware, or deletion by attackers.
Seven components recur across Meta's DelF, Netflix's deletion platform, GitLab's 2025 blueprint, the cloud key and project services, and the remediations the incidents produced. No single system has all seven.
The common shape separates the request to delete from the act of destroying. A request moves the target into a pending state, which hides it, stops it being used, and keeps its bytes. A separate, slow, rate-limited reaper destroys what has sat in that state past a window. Everything that makes deletion safe happens in the gap between those two events, and every incident in section 04 is a case where some path skipped the gap.
The delete names a kind and an identifier, and refuses an identifier of the wrong kind. Atlassian's script accepted a list of IDs and could not tell site IDs from app IDs. DelF annotates every object and edge type with how deletion should propagate, so the framework, not the caller, decides what goes with what.
Seen at: Meta DelF; absent at Atlassian 2022
Before anything moves, compute what the request covers, including children, and compare it with a floor or a ceiling. AWS's post-2017 tool refuses removals that would take a subsystem below its minimum capacity. Kubernetes' interactive delete previews the resolved list rather than asking per item.
Seen at: AWS S3, Kubernetes KEP-3895
The undo window lives on the unit one command can destroy: a project, group, key, vault or site. GitLab moves "the item and all its child items" into pending deletion together; AWS KMS holds keys in PendingDeletion for 7 to 30 days; Google Cloud projects sit in DELETE_REQUESTED for 30.
Seen at: GitLab, AWS KMS, Google Cloud
Destruction runs asynchronously, at a capped rate, below live traffic. Netflix treats deletes as low-priority requests; AWS's remediation made its capacity tool remove servers "more slowly". Slowness is the safeguard: it turns a fat-fingered list into an alert before it becomes an outage.
DelF writes a write-ahead "restoration log" for every deletion and has used it to recover from deletions of "the wrong top-level object" in tens of incidents. Netflix runs journal and recovery services. A journal is what lets you undo a row-level mistake without soft-deleting every row.
UniSuper's private cloud came back because backups in a separate Cloud Storage location "were not impacted by the deletion". Spotify's clusters came back because its backups had already been restored once in a drill. A backup inside the container dies with it.
Seen at: Google/UniSuper, Spotify
The seventh component, a notifier, is the one Google's report makes central. A notification sent when the request is made is useless when the request is a blank field filled in a year earlier. The notification that matters fires on the reaper path, just before destruction, to the owner of the thing, whoever asked. Google does not say whether its corrected system now does this; it says it "corrected the system behavior that sets GCVE Private Clouds for deletion for such deployment workflows" and deprecated the internal tool. That is a fix to the trigger, not to the notifier.
Where the systems diverge is the granularity of the pending state. Meta and Netflix delete billions of rows and journal them; GitLab, AWS, Google and Microsoft put the window on large containers and let rows inside them go. Atlassian's 2022 remediation tried to do both, committing to universal soft deletes so that customer data "that has not gone through a soft-delete process cannot be deleted globally". Section 03 argues the divergence is not taste; it follows from which unit a single command can destroy.
Five forks, each with the option that lost and the condition that would make it win.
The row-level argument is the best documented disagreement in this corpus, and it resolves cleanly once you ask what the soft delete is for. Brandur Leach's 2022 essay, written from Heroku and Stripe experience, says the deleted_at column costs a predicate on every query and the protection of foreign keys, and that in more than ten years nobody he worked with used it to undelete anything. The discard gem's README is written by someone who maintained the most popular Rails soft-delete library and concluded "it does the wrong thing for most cases", because a soft destroy still fires dependent destroys on the children. Django closed the request to own soft delete as wontfix in 2011 (#16567), and in 2022 closed a bug where a soft-delete default manager made the admin unable to restore rows (#33375). Three ecosystems, one finding: a row-level soft delete that is woven into every read path breaks the restore path it exists to serve.
Atlassian went the other way, and for a reason the essay does not cover. Its stores were multi-tenant, so restoring one customer meant extracting their rows from a point-in-time copy that also held everyone else's, which is why restoration ran in batches of up to 60 tenants and took four to five elapsed days each. The rule I draw from both: put the undo window on the unit a single command can destroy. If that unit is a project, soft-delete projects. If one command can remove a tenant's rows across shared tables, those rows need either a tenant-level pending state or a journal keyed by tenant, and a restore tool that works at that granularity.
These look like opposite philosophies and are not. GitLab's argument was that aliasing
rm protects against one command and "not against disk corruption or any of the many
other ways you can lose data", so the durable investment is a restore you trust. That is right for
data. Cloudflare's R2 incident and AWS's S3 incident destroyed nothing; they removed serving
capacity, and the outage lasted exactly as long as it took to put it back. For those, the only
lever is to stop the operation or slow it, which is what both remediations did. A design review
should classify each destructive operation by which of the two it destroys before choosing the
safeguard.
kubectl delete --interactive, stable in v1.30, shows the resolved list once--all, "historically determined ... a breaking change"The Kubernetes record is worth reading for its honesty about why the safe default never
shipped. The kubernetes-dev RFC says confirmation for --all was "a long requested
feature" that the project kept declining because scripts would break. KEP-3895 cites "mistyping,
hasty tab completions" as the motivating failures and settles for an opt-in flag. Its rejected
alternative is the instructive part: a yes-to-all prompt was turned down because it asks for
consent without showing scope. A prompt that does not display the resolved target list is not a
safeguard; it is a speed bump that trains the operator to type y.
GitLab's design document is unusually frank about the cost. Pending items "still count towards usage quotas", and the rejected alternative, which would have stopped users purging early, was priced at roughly four times the storage and would have left users "unable to immediately and permanently remove sensitive data without admin involvement". GitLab took the longer window but kept user purge. The window length is really a detection budget. Atlassian's customers noticed within hours; Google's customer could not have noticed a deletion that was scheduled a year earlier by a default nobody knew existed. No window helps the second case. Only the notifier does.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Undo granularity | Container pending state; journal for rows | Default-scoped deleted_at | Breaks foreign keys, restore path, child cascades | Brandur 2022, discard |
| Guard or recover | Recover for data; guard for availability | One policy for both | Recovery does not shorten an availability outage | GitLab 2017, Cloudflare 2025 |
| Bulk confirmation | Opt-in preview of resolved list | Default prompt; yes-to-all | Compatibility; consent without scope | KEP-3895 |
| Window length | 30 days, user may purge early | Admin-only purge | ~4x storage; privacy of early removal | GitLab design doc 2025 |
| Cascade order | Workloads before the policies guarding them | Semi-random namespace teardown | Pods outlived their NetworkPolicies | KEP-5080 |
Seven incidents in four classes. The class matters more than the company, because the class tells you which checkpoint in Figure 1 to fix.
Four incidents, four organisations, one shape. The intended target and its container were both valid inputs to the same operation, under the same credential, and nothing compared the size of what was about to happen with the size of what was meant.
Two incidents where the target was the right kind of thing in the wrong place. The fix in both lives in the operator's environment rather than in the API.
rm.Nobody asked for a delete. A default asked for it. This is the class the request-path safeguards cannot see, because there is no request.
Terraform's documentation describes the same class as intended
behaviour. Its prevent_destroy lifecycle flag "does not prevent the remote object
from being destroyed if the resource block were removed from configuration entirely: in that
case, the prevent_destroy setting is removed along with it." A protection stored inside the thing
it protects disappears with it. The cloud services avoid this by storing the protection on the
service side: Key Vault retention "can't be changed" once saved, and purge protection cannot be
turned off.
Spotify's second deletion is the only published case in this corpus, and it is enough to make the point. A month after the first incident the team was "trying to prevent accidental cluster deletions by codifying them"; review builds "unknowingly modified global state", and "two PRs merged out of order". Recreating the cluster then failed for lack of permissions, the team granted "enough but different permissions", and that changed Terraform's view of the clusters. The safeguard was a new path to the same delete, and it was less tested than the old one. The same pattern appears in a different domain in this series' guide on retry storms, where AWS's remedy for a 2014 outage sustained a 2015 one. Treat every deletion safeguard as a new destructive code path that needs its own test.
The figure that matters most is not in any single source: deletes run at thousands of times the speed of the restores that undo them.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Sites deleted by one script run | 883 in 23 min | Atlassian | 07:38 to 08:01 UTC | 2022-04 | PIR |
| Restore batch and elapsed time | 60 sites, 4 to 5 days | Atlassian | Before parallel batches | 2022-04 | update |
| Delete-to-restore speed ratio | ~4,000x (estimate) | Atlassian | 38 sites/min deleted vs 60 sites per ~4.5 days restored; derived | 2022-04 | derived from PIR and update |
| Outage after server-level delete | 10 h 21 min | Azure DevOps | 17 databases; detected in 20 min | 2023-05 | DCD |
| Gateway disabled by abuse action | 59 min | Cloudflare R2 | No data lost | 2025-02 | Cloudflare |
| Capacity removal to placement recovery | ~4 h 17 min | AWS S3 | 9:37 to 1:54 PM PST; derived | 2017-02 | AWS |
| Data removed before the wipe was stopped | ~300 GB | GitLab | "a second or two" of rm | 2017-01 | postmortem |
| Cluster restore time | 3.25 h | Spotify | 50-node GKE cluster, tested backups | 2019-05 | slides |
| Silent default before deletion | 1 year | Google GCVE | Blank parameter | 2024-05 | |
| Deletions executed | billions per day | Meta DelF | Three social networks | 2020-08 | Meta |
| Mis-annotated edge types | ~2 per day | Meta DelF | Over-deletion 2x over-retention | 2020-08 | paper |
| Rows deleted by central platform | 76.8 billion | Netflix | 1,300 datasets, zero data loss (presenters' figures) | 2025-11 | InfoQ |
| Pending-deletion window | 7 to 30 days | GitLab.com | Changed 10 July 2025; ~4x storage for the rejected variant | 2025-07 | GitLab |
| KMS key waiting period | 7 to 30 days, default 30 | AWS | Cancellable while pending | 2026 | AWS docs |
| Key Vault retention | 7 to 90 days, default 90 | Azure | Fixed once set; mandatory since Feb 2025 | 2025 | Azure docs |
| Project recovery period | 30 days | Google Cloud | Cloud Storage and Pub/Sub delete "much sooner" | 2026 | Google docs |
The asymmetry row is my arithmetic and should be read as an order of magnitude. Atlassian's script removed 883 sites in 23 minutes, about 38 a minute. Its restore, before it learned to run batches in parallel, returned 60 sites every four to five days, about one site every two and a half hours. The ratio is in the thousands. Azure shows the same shape in a smaller system: one API call, ten hours back. Any capacity plan for recovery that assumes restore is roughly as fast as delete is wrong by three orders of magnitude, and the reason is structural: deletion needs only the identifier, while restoration needs the identifier, the bytes, the configuration, the permissions and every system that referenced the thing.
The Azure DevOps figures come from three trade outlets quoting Microsoft's postmortem, which this session could not retrieve; they agree with each other. Atlassian's customer count varied between its own documents (775 in the PIR, "approximately 400" in an earlier update); the PIR is later and is used here. Netflix's 76.8 billion is the presenters' figure as reported by InfoQ. No public source gives the cost of a pending-deletion window in currency; GitLab's "fourfold" is a relative storage figure for one rejected design.
What nobody has published. No organisation in this corpus reports how often its undo window is used on the operator path versus the customer path, which is the number that would test this guide's main claim. No postmortem reports a destructive operation that a pending state caught in production; the systems that have one publish their design, not their saves. DelF's "tens of incidents" is the closest, and it does not break them down.
Every source behind this page, graded. Sources marked [search] sit on hosts this
session's network denied and were read through retrieval; see sources.md.
A decommission script ran with site IDs instead of app IDs and in permanent mode; 883 sites deleted; restore tooling built for single sites.
A mistyped playbook input removed index and placement servers; the fix slowed the tool and gave it a capacity floor, then audited every other tool.
A blank parameter in an internal tool assigned a one-year term; the private cloud was deleted without notification because the customer had not asked.
Wrong-host wipe of the primary; GitLab chose recoverability and clearer prompts over blocking commands, and said why.
An abuse-remediation action meant for one phishing endpoint disabled the R2 Gateway for 59 minutes.
Reports Microsoft's postmortem: the snapshot clean-up job deleted the SQL server and its 17 databases; 12:10 to 22:31 UTC. The primary was not retrievable in this session.
Names the change: deprecated Microsoft.Azure.Management packages replaced by Azure.ResourceManager.
The prompt truncated the hostname to "db1" and used identical colours on staging and production.
Unifies pending deletion across tiers and namespace types; children follow the parent; rejects admin-only purge at about four times the storage.
Protection against accidental deletion had been a paid-tier feature; this epic extended it to free projects and groups to cut recovery tickets.
Closed 10 July 2025; lists storage impact analysis and performance testing as requirements.
Default confirmation ruled out by compatibility; opt-in preview shipped stable in v1.30; yes-to-all rejected as consent without scope.
Namespace teardown was semi-random, so pods could outlive the NetworkPolicies guarding them; now pods go first. Stable in v1.34.
States that confirmation for --all was long requested and repeatedly declined as a breaking change.
A former paranoia maintainer: default-scope soft delete "does the wrong thing for most cases", and soft destroys still cascade dependent destroys.
The framework declined to own soft delete; a soft-delete default manager later blocked restores in the admin.
Annotated deletion semantics, static and dynamic validation, and restoration logs used in tens of recoveries; measures the human error rate in cascade rules.
A 597-outage corpus from public reports, 2009 to 2015, classifying root causes and fixes. Used here only to establish that the broad record exists; its deletion-specific breakdown was not retrieved.
Two production cluster deletions; the second caused by the attempt to codify against the first; survived by partial migration and tested backups. Claims cited by slide number.
QCon SF 2025, 49:41: deletion as a platform with audit loops, tombstone control and resurrection detection.
Names ad-hoc human production changes as an outage class ("I never make tipos") and argues every change be automated, pre-validated, or break-glass.
Billions of deletions a day, each with a restoration log that allows undo "for a limited time period".
deleted_at costs a predicate on every query and the foreign keys; never used to undelete in ten years; proposes a deleted_record table.
Follow-up: the author's team moved off deleted_at except on a couple of core tables.
Delayed deletion made default for paid tiers in 16.0 because support kept recovering accidental deletions.
30-day window live on GitLab.com from 10 July 2025; admin-initiated deletions still to be routed through it.
Restore in batches of up to 60 tenants, four to five elapsed days per site.
Four months on, the first disaster-recovery exercise for site deletion was still scheduled for the last quarter of 2022.
The dissenting read: Atlassian published a postmortem but would not give the number of affected users.
Soft (lazy) deletion as the first defence layer, ahead of backups and validation.
prevent_destroy is removed along with the resource block it protects.
No immediate delete exists: a 7 to 30 day waiting period, cancellable, key unusable while pending.
30-day recovery for projects, but Cloud Storage and Pub/Sub inside them may be deleted much sooner.
Retention 7 to 90 days, fixed once set; soft delete mandatory on all vaults since February 2025 per a Trend Micro note.
Absent tier: no closed-unmerged pull request on deletion safety was retrievable through this session's network policy. The nearest rejected-approach records are Django's wontfix tickets, KEP-3895's rejected alternatives, the kubernetes-dev RFC and GitLab's rejected admin-only purge.
Six rungs, from a table with a pending state to a deletion path you have attacked on purpose.
In a toy service with projects and child records, make DELETE move a project to a pending_deletion state with a scheduled purge time. Children are hidden through the parent, not by their own flag.
Done when: a deleted project's children vanish from every read and come back intact on restore. Teaches: why the window belongs on the container, not on each row.
Write the purge as a separate job that selects only items past their window, at a capped rate. Remove every other code path that can hard delete.
Done when: grep finds exactly one hard-delete call site. Teaches: Atlassian's lesson: the permanent mode should not accept caller input.
Give project and organisation IDs different prefixes, reject the wrong kind, and make every bulk delete print the resolved count and refuse above a ceiling.
Done when: passing an org ID to the project delete fails before any write. Teaches: scope inflation is catchable at the type, not in review.
For rows deleted inside a live project, copy them as JSON to a deleted_record table keyed by tenant before removal, with expiry.
Done when: you can restore one tenant's deleted rows without touching anyone else's. Teaches: the restore granularity Atlassian lacked.
Send the owner a notice when the reaper is about to purge, regardless of who requested it. Then delete something through your admin tool, a migration and a clean-up job, and check that each lands in pending state.
Done when: no path, including your own scripts, reaches Destroyed without passing Pending and a notice. Teaches: Google's finding that safeguards follow the caller unless you move them.
Time a bulk delete of 1,000 containers and the restore of all 1,000. Then run a game day: swap an identifier list, blank a retention field, upgrade the client library under the clean-up job against a fixture with real targets.
Done when: you have a measured delete-to-restore ratio and each injected fault was caught before the reaper. Teaches: what your recovery capacity really is, which no design review tells you.
The queries that found the material above. The useful vocabulary is "pending deletion", "restoration log", "delayed deletion" and the clean-up job's own name.
post-incident review script deleted sites "mark for deletion" permanently deleteoutage postmortem snapshot deletion job deleted "SQL server"incident abuse remediation disabled service instead of endpoint"remove a small number of servers" "larger set" postmortem"accidentally deleted" production cluster keynote slides"pending deletion" design document "alternative solutions" storageKEP kubectl delete interactive "backwards compatibility""delayed deletion" epic site:gitlab.com"ordered namespace deletion" NetworkPolicy KEP"soft deletion" "isn't worth it" deleted_at foreign keys"Why not paranoia" default_scope soft deletesite:code.djangoproject.com soft delete wontfix managerprevent_destroy "removed from configuration""restoration log" deletion framework write-ahead undocentralized data deletion platform tombstones resurrection talk"lazy deletion" "soft deletion" data integrity defence in depthScheduleKeyDeletion waiting period PendingDeletion