In April 2022 an Atlassian maintenance script permanently deleted 883 sites belonging to 775 customers in 23 minutes, because a request passed between two teams did not say which identifiers or which deletion mode were meant. What does this say about written instructions as an engineering interface, and what would you change?

The situation they were in

One team was retiring a legacy application. Another team owned the script that could remove it. The published post-incident review describes the mechanism plainly: the script supported both "mark for deletion", used routinely, and "permanently delete", required for compliance erasure, and it was run with the wrong mode against the wrong list of identifiers. Deletion ran from 07:38 to 08:01 UTC on 5 April 2022. The first customers were restored on 8 April and the last on 18 April, so the worst-affected customers were down for around two weeks.

What the incident actually turned on

The hand-off was a written request, and the written request was the interface. It carried identifiers that could refer to either an application or a whole site, and a verb whose two meanings differed by whether the data still existed afterwards. Neither team's software was wrong. The ambiguity lived entirely in the sentence.

That is the uncomfortable transferable point: for a large class of operational work, the specification with the highest blast radius in the company is a paragraph in a ticket, written in a hurry by one engineer and executed literally by another.

What to change, in order

1. Make destructive and reversible operations different verbs in different tools. If one flag separates "recoverable" from "gone", the flag will eventually be wrong. Permanent erasure should be a separate program with a separate approval path, not a mode.
2. Make the request machine-readable. A ticket that hands over a list of identifiers should hand over a file with a type declared — site ids or application ids — validated by the tool before execution. Prose is for reasons; parameters belong in a format that can be checked.
3. Require the executing team to restate the effect and the count before running it. "This will permanently delete 883 sites across 775 customers, unrecoverable" is a sentence that would have stopped this. It costs one message.
4. Put a blast-radius limit in the tool. Above a threshold, require a second approver and a staged run. The script had no reason to be able to remove hundreds of customers in one invocation.
5. Test the restore path at the granularity you will need it. Atlassian's systems were designed to restore the whole environment, and the incident required restoring individual sites into a running multi-tenant estate, which is a different and much slower operation.

When this is the wrong lesson

The conclusion "add approvals to everything" produces a queue and a culture of rubber-stamping, which makes the next incident more likely rather than less. Tie the ceremony to the reversibility and the reach: an operation that is recoverable in minutes needs none of this, and an operation that permanently destroys customer data needs all of it regardless of how routine it feels.

What a strong answer adds

The communication lesson extends past the script. Restoring 775 customers over two weeks is mostly a communication problem: the affected customers cannot log in to a status page, support cannot tell them a date because restores are sequential and uneven, and the company's public channels are the only route left. The published review discusses the difficulty of reaching affected customers during a restoration that ran for days, which is worth reading before designing anybody's incident communication plan.