The delete lands one level up, on the path nobody guarded
How production systems design destructive operations (deleting tenants, resources, data and capacity) so that a wrong target is caught or undone, read from seven published deletions and the deletion designs of Meta, Netflix, GitLab, Kubernetes and the cloud key and project services.
Reconstructs the safe-delete architecture from the Atlassian 2022, Azure DevOps 2023, Google Cloud GCVE 2024, Cloudflare R2 2025, AWS S3 2017, GitLab 2017 and Spotify 2019 incidents, Meta's DelF paper, Netflix's 2025 deletion platform, GitLab's 2025 deletion design document and Kubernetes KEPs 3895 and 5080. After reading, an architect can say where their undo window lives, whether their own operators' scripts and clean-up jobs pass through it, which destructive operations need a guard rather than a recovery path, and how much slower their restore is than the delete that would need it.
None of the seven published deletions was a customer deleting their own data: every one ran on the operator's housekeeping path, and the undo machinery (notification, soft delete, self-service restore) was attached to the customer's path; Google's report says outright that a customer-initiated deletion would have been preceded by a notification.
What you get out of it
- Safeguards follow the caller unless you move them: route every operator script, clean-up job and admin tool through the same pending state, and put the notification on the reaper path where it fires for deletions nobody requested.
- Scope inflation is the dominant failure class: in four of seven incidents the delete hit the container of its target (app to site, database to server, endpoint to service, a few servers to a fleet), because parent and child IDs are the same shape and share a credential.
- Put the undo window on the unit one command can destroy: containers get a pending state, rows get a journal; default-scoped row soft delete breaks foreign keys and restore paths in Rails, Django and Brandur's account alike.
- Recovery bounds a data delete but not an availability delete: Cloudflare's 59 minutes and AWS's four hours destroyed no data, so those operations need a rate limit, a floor, and removal from roles that do not need them.
- Deletes run thousands of times faster than restores: Atlassian removed 883 sites in 23 minutes and returned them 60 at a time over four to five days, so recovery capacity has to be planned, measured and drilled separately.
Scope
Why this, now. GitLab rebuilt its whole deletion flow in 2025 and published the alternative it rejected, Microsoft made Key Vault soft delete mandatory in February 2025, Cloudflare disabled its own R2 gateway the same month while removing one phishing bucket, and Netflix described its deletion platform at QCon in November 2025.
What it does not cover. Backup and restore drills (covered by an earlier guide), privacy-law erasure obligations, ransomware and attacker-driven deletion, and the internals of tombstones in LSM storage engines.
Other field guides
Evacuating the region: who can actually leave, and what it costs
Reconstructs regional evacuation from GitLab's public DR blueprints, runbooks and gameday timing ledger, Meta's Maelstrom and Taiji papers, Netflix's…
26 sources · 15 organisations · 6 postmortemsComing back from cold: the restart is a harder load case than the crash
Reconstructs, from the Kubernetes project's own design records, the fixes HashiCorp shipped after October 2021, the Chubby and Physalia papers, Netfl…
20 sources · 10 organisations · 4 postmortemsThe way back is also a deploy
Reconstructs rollback as it actually works in production: a forward deploy of an old artifact, underwritten by a compatibility contract fixed weeks b…
28 sources · 25 organisations · 6 postmortems