On 5 April 2022 an Atlassian maintenance script deleted 883 customer sites belonging to 775 customers in about 23 minutes. Restoration ran until 18 April, up to 14 days for some customers, although no customer lost more than about five minutes of data. What failed, and which design decisions made that gap between deleting and restoring possible?

The trigger

Two teams were coordinating the deactivation of a legacy app. One team supplied identifiers; the other ran a script. The identifiers were site ids rather than app ids, and the script did not cross-check which kind it had been given. It called a deletion path that removed entire sites rather than an application within them.

The whole deletion ran between roughly 07:38 and 08:01 UTC. Nothing in the path required a second person to confirm the target, and nothing forced the delete to be recoverable.

Why the recovery took so long

This is the instructive part, and it is a design consequence rather than an operational failure.

• Backups existed and were good. Data loss was minimal — five minutes at most. The problem was never durability.
• Restore granularity did not match the incident's shape. The tooling was built for restoring a whole environment after a large-scale event, not for returning several hundred individual sites into a running multi-tenant environment where other customers were live and unaffected.
• Per-site restoration therefore had to be done partly by hand, verified per customer, and sequenced. Hundreds of individually verified restores take days regardless of how much automation exists for a different scenario.

The structural fixes, in priority order

1. Make destructive operations soft by default. Mark for deletion with a delay measured in days, and require an explicit second step to make it permanent. This turns the whole incident into a recoverable mistake within minutes.
2. Validate the type of what you are given. The script received an identifier and did not check what kind it was. A type check costs a line of code and would have prevented this.
3. Two-person integrity on bulk destructive actions, not on everything: any action affecting more than a handful of tenants needs a second approval. This is segregation of duties applied where it earns its cost.
4. Rehearse the restore shape you will actually need: single tenant, into a live environment, under time pressure. A full-environment restore drill does not exercise it.

The general lesson

Backups answer durability; they do not answer granularity. The question that separates the two is "what is the smallest unit I can restore, into a running system, without touching anyone else?" Teams discover the answer during an incident unless they ask it during design.

Common weak answers

• "They should have tested the script." In a staging environment with correct ids it would have passed. The missing controls are the soft delete and the type check, not more testing.
• "Take more frequent backups." Recovery point was not the problem; recovery granularity was.
• "Remove the deletion API." Something must be able to delete. Make it reversible for a window, and make the irreversible version a separate, guarded operation.