In April 2022 a maintenance script at Atlassian permanently deleted 883 sites belonging to 775 customers inside 23 minutes, and the last customer was not restored until 18 April. Backups were fine and almost no data was lost. What made recovery the hard part, and what should that change in how you evaluate a vendor?

The situation they were in

Atlassian's published incident review describes a script that supported two modes, "mark for deletion" used in routine operations and "permanently delete" required for compliance. During the removal of a legacy app, one team supplied site identifiers where app identifiers were expected and the script ran in the permanent mode. Between 07:38 and 08:01 UTC on 5 April 2022, 883 sites were deleted.

The backups did their job: no customer lost more than five minutes of data. The first customers were restored on 8 April and the last on 18 April, up to 14 days after the deletion.

Why recovery was the hard part

Their backups were designed for the failure they had planned for — losing everything — not for the failure they had: losing an arbitrary subset of tenants from a shared estate. Restoring the whole platform to a point in time was automated and fast. Extracting one tenant's rows from dozens of multi-tenant services, reassembling them into a new site, and reconnecting the identities, attachments and cross-product links was largely manual at the start.

That is the transferable lesson: recovery granularity is a property of its own, independent of how good your backups are. RPO measures how much you lose. Nothing in an RPO number tells you whether you can restore one customer out of a million without touching the other 999,999.

What to ask a vendor instead

• "Show me a timed single-tenant restore into a new site, performed last quarter." Not a policy, a run. Ask what the elapsed time per tenant was, and whether it was concurrent or serial.
• "How do you enumerate one tenant's data across every service?" If the answer is a list of service owners, your restore is a coordination problem and it will take days.
• "What is irreversible in your admin tooling, and what is the delay before it becomes so?" A permanent delete that runs with no cooling-off period is the risk, not the script that calls it.
• "How do you reach affected customers when their site — and so their contact record — is gone?" Atlassian's review is explicit that the notification path assumed the site existed.

Where copying this would be a mistake

Do not build tenant-level restore tooling because a large vendor needed it. For most products the proportionate control is upstream: make destructive operations soft by default with a recovery window of days, require a second identity to escalate to permanent, and cap blast radius per invocation. That costs a sprint. Tenant-granular restore across a service estate costs a programme, and it is justified when a subset failure is plausible and a full rollback is not: multi-tenant shared storage, many customers, independent recovery obligations.

Common weak answers

• "They needed better backups." They had good backups. The gap was restore granularity and automation.
• "Add approval to the script." Necessary and insufficient: the operator had the right approval for the operation they believed they were running.