Review Scope Mismatch

A script is reviewed. The reviewer checks the endpoint being called, the parameters, the error handling. It all looks right, and it is right. The script then deletes hundreds of production tenants, because nothing in the review asked what the identifiers in the input file referred to.

The reviewer did not fail. The control was pointed at the wrong property. That distinction matters, because organisations respond to the incident by strengthening the control - more reviewers, more sign-offs - which strengthens the inspection of a property that was never the risk.

Why it matters

A control that fails while reporting success is the dangerous kind. A control that fails loudly gets fixed; one that passes honestly and lets damage through creates confidence that suppresses the search for a real safeguard. Every subsequent risk assessment now has a green tick next to "peer reviewed".

The second reason is that the usual remedy makes things worse in a measurable way. Adding reviewers raises review latency, which makes people batch changes to amortise the wait, which increases the blast radius of each run - the exact variable that caused the harm. And responsibility diffuses: two reviewers each apply less scrutiny than one alone, because each assumes the other covered what they skimmed.

Implementation patterns

• Make the change state its own blast radius before it acts. How many tenants, of what type, sampled and resolved to names. A reviewer looking at "this will permanently delete 883 production sites" behaves differently from one looking at a well-formed API call.
• Dry run as a required artefact, with its output attached to the review rather than described in it.
• Bound the batch. A destructive operation that cannot exceed N targets per invocation converts a catastrophe into an incident, and it needs no human judgement to work.
• Type the dangerous arguments. An API that accepts an identifier without knowing what kind of entity it identifies is the root defect; distinct types for app IDs and site IDs would have failed the call at the boundary.
• Soft delete with a delay before permanent removal, so the worst outcome is a restore rather than a reconstruction.
• Match review depth to reversibility, not to code size. A one-line script with an unbounded target set needs more scrutiny than a thousand-line refactor behind a flag.

Industry example

Atlassian's published post-incident review of April 2022 records that 883 sites belonging to 775 customers were deleted between 07:38 and 08:01 UTC on 5 April, during work to deactivate a legacy app. The script had followed the standard peer-review process, which focused on which endpoint was called and how; it did not cross-check whether the supplied cloud site IDs referred to the app or to entire sites. The first customers were restored on 8 April and the last on 18 April, up to 14 days later, and Atlassian reported that no customer lost more than five minutes of data.

The gap between "five minutes of data lost" and "fourteen days to restore" is the part worth carrying away: the durability design was sound and the restore granularity was not.

Failure scenarios

• Review of a deploy that references an external data file - the code is reviewed, the file that determines what it touches is not.
• Infrastructure-as-code plans approved without reading the plan output, so a resource replacement reads as an update.
• Migrations reviewed for SQL correctness with no statement of how many rows or which tenants they reach.
• Runbooks whose steps were reviewed individually, with no one asking what the sequence does if step three is run twice.
• Feature-flag changes outside review entirely, where the blast radius is 100% of traffic and the control surface is a toggle.

Trade-offs

Making blast radius visible is real engineering work: the tooling must resolve identifiers, count targets and render them meaningfully, and it must do that for every class of destructive operation rather than once. It also slows down the dangerous operations specifically, which is the point and is still a cost.

The alternative - more approvers - is cheap to introduce and expensive to run forever, and it buys latency rather than safety. The choice is between paying once in tooling and paying continuously in queueing.

When not to use it

When the reviewer can already see the blast radius from the diff. For ordinary application changes behind a progressive rollout, the change is its own description of reach, and an extra reviewer is genuinely cheap insurance that costs nothing meaningful in latency.

Also when the operation is bounded and reversible by construction: if the worst case is a rollback, the effort belongs elsewhere. The discipline is for the narrow class of operations that are destructive, unbounded, or outside the normal deployment path - which is where incidents of this shape come from.

Interview question

Q: An engineer proposes a one-line change to a maintenance script and asks you to review it. What do you ask for before approving, and how would you change the process so the next reviewer does not depend on being as careful as you?

What a strong answer covers: asking what the script will touch and how that is determined - especially any input outside the diff · requiring a dry-run output as part of the review artefact · recognising that process-by-diligence does not scale, so the durable answer is tooling that prints the blast radius and a hard batch bound · typing the dangerous arguments so the mistake is impossible rather than merely caught · and distinguishing this from "add another approver", with the latency and diffusion-of-responsibility costs that carries.

Quick check

Quiz: A peer-reviewed script caused a large deletion. Why is a second reviewer usually the wrong fix? Because the second reviewer inherits the same scope and the same missing information, so the cost doubles and the inspected property does not change - while review latency rises and changes get batched into larger, more dangerous runs.

Flashcard: What is the test for whether more reviewers will help? — Whether a reviewer can see the blast radius from the diff. If yes, more eyes help. If no, the effort belongs in making the radius visible.