In April 2022 Atlassian ran a maintenance script to deactivate a legacy application and, because the wrong identifiers were supplied, it instead deleted the sites of roughly 775 customers. Restoration took up to about two weeks for some of them. The script ran through normal operational tooling. Treating that tooling as part of the delivery system, what failed and which design decisions allowed it?

The trigger

A destructive bulk operation executed with an input that was accepted as valid. The script worked correctly; the identifiers described the wrong objects. No code defect, no infrastructure failure, no bad deployment in the conventional sense.

Why it propagated

Three properties of the tooling, each of which looks reasonable alone.

The operation was bulk and atomic from the operator's point of view. One invocation acted on hundreds of tenants. There was no mechanism forcing the operation to proceed in stages with a pause between them, so the blast radius was decided by the length of the input list rather than by any policy.

The input was identifiers rather than names. Whatever review the invocation received, the decisive control for a destructive operation is that the operator sees the resolved list of objects in human-readable form and confirms it. An identifier is unreadable to a reviewer and a name is not, which is how an error of this kind survives a second pair of eyes anywhere it occurs.

The deletion was permanent rather than deferred, since the published account describes recovery by restore rather than by reversal. A soft delete with a retention window turns an irreversible mistake into a recoverable one, and the recovery is a flag flip rather than a restore.

Why recovery was slow

This is the part that generalises most. Backups existed and were not the bottleneck. The constraint was restore granularity and concurrency: a platform designed to restore everything after a catastrophe is not the same as one that can restore 775 individual tenants, each into a live multi-tenant estate alongside hundreds of thousands of unaffected ones, without disturbing anyone else.

Per-tenant restore had to be assembled under pressure, verified per tenant, and run with limited parallelism. A tested bulk restore path is a product capability, and if it has never been exercised its throughput is unknown — which is why the honest answer to "how long will this take" was days.

The structural fix versus the tempting local fix

Tempting: add a confirmation prompt to the script, and require two people to run it. Both help and neither addresses the mechanism; a second pair of eyes on an identifier list sees the same unreadable input.

Structural, in order of value:

1. Soft delete with a retention window for anything tenant-scoped. This alone converts the incident from a two-week restore into a reversal.
2. Resolve-and-confirm. The tool prints what the input resolves to, in human terms, with counts, and refuses to proceed without an explicit confirmation of that resolved list.
3. A blast-radius cap in the tool itself. Above a threshold — 10 tenants, say — the operation requires a separate authorisation and proceeds in batches with a pause.
4. Operational scripts on the same path as code. Reviewed, versioned, tested against a realistic fixture, and run by the pipeline rather than from a laptop. The reason this belongs in a delivery-pipeline discussion is that a destructive script is a deployment with no staged rollout.
5. Rehearse per-tenant restore. Measure how long 10 tenants take, and publish the implied figure for 1,000.

The general lesson

Administrative tooling usually has the widest blast radius in an organisation and the weakest controls, because it is classified as operations rather than as delivery. A deployment gets canaries, staged rollout, automated rollback and a review; a script that can delete customer data gets a shell prompt.

The test to apply: for every tool that can act on more than one customer at once, what is the maximum damage one invocation can do, and what makes it reversible? If the answer to the second question is "a restore", the answer is not good enough.

When this is over-engineering

For a single-tenant internal system, or one where the operation is additive, this machinery is unnecessary. Soft delete still earns its place almost everywhere because it is cheap, but resolve-and-confirm and blast-radius caps are worth building specifically for multi-tenant, destructive, bulk-capable operations. Those three properties together are the trigger; any two of them are not.