A fraud system routes flagged transactions to human reviewers, who approve or reject within a target of 4 minutes each. A campaign triples flagged volume overnight. Reviewer headcount is unchanged. What happens to the quality of the human oversight, and what should the design do about it?

What happens, in order

Hours 0–2: the queue grows and nothing else changes. Reviewers work at their normal rate. The backlog is the only visible signal, and it is on a dashboard nobody is watching at 2am.

Hours 2–8: handling time collapses. Under a growing queue, reviewers speed up. Average review time falls from 4 minutes to under 1, and the approval rate rises, because approving is the faster action and the model's recommendation is usually right. The control is now a rubber stamp, and its measured throughput looks better than ever.

Day 2 onwards: the oversight costs more than it buys, because everything downstream, including the regulator's view of the process, still records a human decision on every item. An automated decision with a human name attached is harder to challenge than an openly automated one.

Where it amplifies

The model was calibrated against reviewer behaviour. If reviewer approvals feed back as training labels, a week of rushed approvals teaches the model that the flagged cases were fine, and the next version flags fewer of them. The degradation outlives the campaign.

What stops it

Not "add reviewers", which is right and slow. The design has to degrade honestly:

• Shed by risk, not by arrival order. Under load, auto-approve the lowest-risk band outright and queue only the high-risk tail. A documented automatic decision is more defensible than a nominal human one.
• Enforce a floor on review time. If median handling time falls below a threshold — say 60 seconds — the system should slow intake or raise the auto-decision band, because a review shorter than the floor is not a review.
• Decouple the model from reviewer labels under load, so rushed decisions do not become training data.
• Give the reviewer a reversal window: a decision that can be undone within 24 hours turns a rushed approval into a recoverable one.
• Alert on the oversight quality metrics — handling time, approval rate, disagreement rate with the model — not on queue depth alone. Choose the approval rate as the page-worthy signal. Automation bias has been documented in the human factors literature since the 1990s, and it is strongest exactly when the operator is rushed. Queue depth says you are behind; approval rate says the control has stopped working.

When this is the wrong worry

If the human step exists only for customer experience — a courtesy check on an outcome that is not consequential — then degrading to automatic under load is fine and needs no machinery. The design effort belongs where a regulator, a court or a customer's livelihood depends on the review having been real. Naming which of the two you have is the first question, and many organisations have never asked it.