Irreversible Operation Guard

Atlassian's April 2022 incident is the cleanest illustration available. A maintenance script was given site identifiers where application identifiers were expected, called a path that removed whole sites, and deleted 883 sites belonging to 775 customers in about 23 minutes. Restoration ran until 18 April, up to 14 days for some customers, although no customer lost more than about five minutes of data.

The durability of the data was never in question. What was missing was everything between the command and permanence: no type check on the identifier, no second person on a bulk destructive action, and no delay during which the deletion could simply be undone.

An irreversible operation guard is the small set of mechanisms that close that gap.

Why it matters

Destructive operations have an asymmetry that ordinary changes do not. A wrong deployment is rolled back in minutes; a wrong deletion is restored in days, because restoring is a different operation with different tooling, and it is usually tooling built for a different scenario.

The guard converts an unbounded, slow recovery into a bounded, fast one. That is worth more than any amount of care taken over the script, because scripts are run by tired people at 07:38 on a Tuesday.

Implementation patterns

• Soft delete by default, with a delay of days before permanence and an explicit second step to make it final. During the window, undo is a flag change.
• Type-check the target. The script received an identifier and did not check what kind it was. One line of validation prevents an entire class of incident.
• Two-person integrity scoped to blast radius, not to everything: any action affecting more than a handful of tenants or rows requires a second approval. Applied everywhere it becomes noise and is worked around.
• Cap the batch. A tool that can only act on 10 entities per invocation turns a catastrophe into a nuisance, and the cap is trivial to implement.
• Dry run with a diff, printing exactly what will be affected, with the operator confirming the count.
• Rehearse the restore shape you will actually need: one tenant, into a live environment, under time pressure. A full-environment drill does not exercise it.

Industry example

Beyond Atlassian, the pattern appears wherever platforms handle deletion at scale: cloud providers apply retention windows to deleted resources, and object stores offer versioning and delete markers rather than immediate removal, precisely because immediate permanence has no upside for the provider and unbounded downside for the customer. The 2022 incident is valuable because the company documented the restore-granularity problem explicitly, which is the part organisations usually discover privately.

Failure scenarios

• A tool that deletes and a restore path that only restores everything, so a per-tenant recovery is manual.
• Soft delete with no expiry job, so "deleted" data accumulates and quietly breaks a retention obligation.
• Two-person approval performed by two people on the same shift, one of whom approves without looking, which is a rubber stamp with a second name on it.
• A cascade past the guard: the guarded operation triggers an unguarded downstream deletion in a search index or an event consumer.
• Soft delete that still removes access immediately, so customers experience an outage even though the data is recoverable.

Trade-offs

Soft deletion costs storage and adds a state that every query, export and retention job must handle; a deletion that lingers for 7 days is a deletion that a regulator may ask about. Two-person approval costs throughput and, applied indiscriminately, creates a queue people learn to route around. The guard should be sized by blast radius, so routine single-entity deletions stay fast and bulk operations carry the cost.

When not to use it

Where the data must be destroyed immediately for legal reasons, a soft-delete window is the wrong design and the guard has to move earlier, to the approval of the request itself. Equally, on a single-tenant internal system with a nightly snapshot and no customer impact, the full apparatus is disproportionate: a confirmation prompt and a good backup are enough.

Interview question

Q: You own a multi-tenant platform. Design the guard rails for the internal tooling that can delete a tenant, and tell me what you would rehearse before you trust them.

What a strong answer covers: soft delete with a stated window and a separate permanence step; type validation and batch caps in the tool itself; two-person approval scoped to bulk actions; what happens to downstream systems that consume deletion events; the restore drill being single-tenant into a live environment, timed, with the result measured; and naming the retention obligation that the soft-delete window interacts with, because that is the trade-off a regulator will ask about.

Quick check

Quiz: Atlassian's 2022 restores took up to 14 days although data loss was about five minutes. Which property was missing? — Restore granularity: the tooling restored environments, not individual tenants into a live estate.

Flashcard: Which single change would have contained that incident fastest? — Soft delete by default, making the whole event an undo within the delay window.