In April 2022 a maintenance script at Atlassian used the wrong identifiers and deleted sites belonging to about 775 customers. Restoration took up to about two weeks for some of them, even though backups existed and were working. What property of the recovery design accounts for that gap?

The situation they were in

A script intended to remove a deprecated application was given identifiers that referred to whole sites, and it executed permanently rather than reversibly. Backups existed and were current. The data was not lost; it was unreachable at the granularity in which it needed to be returned.

What made restoration slow

Backups for a large multi-tenant platform are designed around the failure they expect: losing a store, a shard or a region. They are therefore taken and restored at that granularity - restore the whole store to a point in time, which is fast and well practised.

Deleting 775 individual customers out of a shared store is a different shape of problem. Restoring the whole store would roll back every other customer on it, which is unacceptable, so each affected customer must be extracted from a backup and reinserted into a live system that has moved on, with their identifiers, their integrations and their cross-service references intact. That path existed in principle and had never been run 775 times in parallel.

Three multipliers then apply. Automation did not exist for the rare path, so much of it was manual or had to be built during the incident. Verification is per customer, because returning the wrong data to the wrong tenant is a worse outcome than a slow restore. And the work is serialised by people, not by machines, so it scales with responders rather than with capacity.

What it cost them

Weeks of recovery for some customers, a large engineering mobilisation, and a public account that is now the standard reference for this class of failure. The direct lesson is not about the script.

When copying this response would be wrong

The instinct after reading this is to demand per-customer point-in-time restore for every system. For most organisations that is disproportionate, and half-built granular restore is worse than none because it creates a belief.

The transferable questions are cheaper than the capability:

1. At what granularity can you actually restore, and how long does one unit take? If the honest answer is "the whole store", then any incident affecting a subset has a recovery time nobody has measured.
2. Is deletion soft first? A deletion that marks and sweeps after 30 days converts the entire incident into an undo. This is the single highest-value change available here and it costs very little.
3. Do destructive administrative tools take a list they have validated and a dry run they show to a human? The trigger was an identifier type mismatch that no validation caught.
4. Has the rare path been rehearsed at scale? Restoring one customer proves the mechanism; it does not prove the throughput. The recovery time for 775 customers is not 775 times the time for one - it is worse, because coordination and verification do not parallelise cleanly.

The general lesson

Blast radius and restore granularity are the same design decision seen from two sides. A tool that can affect 775 tenants in one action requires a restore path that works at the same scale and has been exercised there. Where that is not affordable, the answer is to prevent the action from having that reach: scoped credentials for administrative tooling, mandatory dry runs, and soft deletes with a window long enough for a human to notice.