intermediate 2 min answer

In April 2022 an Atlassian maintenance script deleted 883 sites belonging to 775 customers in 23 minutes. The script had passed peer review - the review checked which endpoint was called and how, not whether the supplied identifiers referred to an app or to a whole site. What happens if you respond by requiring a second reviewer?

atlassiancode-reviewblast-radiussecond-order-effectscontrols
Show the full answer Hide the answer

What happens, step by step

Nothing improves, and the cost is paid immediately. The first reviewer did their job against the scope they understood the review to cover. A second reviewer inherits the same scope, the same absent checklist and the same missing information about what the identifiers mean. Two people now confirm the endpoint and the call shape. You have doubled the cost of the control without changing what the control inspects.

Then the second-order effects arrive:

  • Review latency rises, so scripts are batched into larger runs to amortise the wait. Larger runs mean larger blast radius, which is the exact variable that hurt here — Atlassian's deletions ran from 07:38 to 08:01 UTC and restoration took until 18 April for the last customers.
  • Diffusion of responsibility. Two reviewers on a change each apply less scrutiny than one reviewer alone, because each assumes the other is covering the parts they skim. This is well documented in incident literature and matches how people behave under time pressure.
  • The control now looks stronger on paper, so the organisation stops looking for the real fix. A control that fails safely is recoverable; a control that fails while reporting success is the dangerous kind.

The actual failure

The review scope did not match the blast radius. Reviewing the mechanics of an API call tells you the call is well formed; it tells you nothing about how many customers the arguments reach. The information the reviewer needed — "these 883 IDs are site IDs, not app IDs, and deleting them is permanent" — was not in the change under review, and no amount of additional reviewers puts it there.

The structural fixes, in order

  1. Make the script print its own blast radius before acting: how many tenants, of what kind, and a sample resolved back to customer names. A human reviewing "this will permanently delete 883 production sites" behaves differently from one reviewing a well-formed call.
  2. Soft delete by default, with permanent removal on a delay. Atlassian's customers lost under five minutes of data; the two weeks were restoration mechanics, not data loss. Recovery granularity is a design decision made long before the incident.
  3. Bound the batch. A destructive operation that cannot exceed N tenants per run converts a catastrophe into an incident.
  4. Require the dangerous class of operation to name its target type explicitly — an API that accepts an ID without knowing what kind of thing it identifies is the root defect.

When this is the wrong answer

For changes whose worst case is bounded and reversible, a second reviewer is genuinely cheap insurance and the latency does not matter. The test is whether the reviewer can see the blast radius from the diff. If they can, more eyes help. If they cannot, more eyes are theatre and the effort belongs in making the radius visible.