Automation Ratchet

In a pipeline where a model handles the easy cases and humans handle the rest, improving the model does not make the human's job easier — it makes it harder. The automated share grows by absorbing the cases that were simplest, leaving a residue that is progressively more difficult, ambiguous or unusual.

Reviewer throughput therefore falls and reviewer error rates rise, while the system as a whole has improved.

Why it matters

Measuring reviewer performance without accounting for it penalises people for the system's success, which is the most common analytical mistake in these pipelines and is corrosive to the people doing the work.

It also invalidates capacity planning based on historical throughput: a review team sized on last quarter's rate will be under-capacity this quarter for reasons unrelated to their performance.

Implementation patterns

• Track case difficulty explicitly — model confidence, ambiguity signals, historical disagreement rate — and normalise throughput and accuracy against it.
• Compare reviewers on comparable cases, using seeded gold-standard items of known difficulty rather than on aggregate output.
• Re-baseline capacity planning whenever the automation threshold moves, since the two are directly coupled and are usually managed by different people.
• Expect and plan for rising cost per reviewed item, and evaluate the pipeline on total cost per unit of work rather than on the human step's efficiency.
• Feed human decisions back as training data, since the residual cases are precisely the most valuable examples — this is the mechanism by which the ratchet advances rather than stalling.
• Recognise the limit. Some residue is irreducibly ambiguous and no model improvement removes it; treating a stable floor as a performance problem chases something that is not there.

Industry example

Large-scale annotation and human-in-the-loop platforms such as Scale AI operate this dynamic as their core economics, and it also appears in content moderation, fraud review, claims adjudication, support triage and document processing — anywhere a confidence threshold divides work between a model and a person.

Failure scenarios

• Reviewer metrics unadjusted for difficulty, producing unfair assessment and attrition.
• Capacity planned on historical throughput after the threshold moved.
• Human decisions not captured as training data, so the residue never shrinks.
• The ambiguous floor treated as a solvable problem, consuming effort indefinitely.
• The automation threshold raised during a backlog without measuring the quality cost, which is a legitimate action made illegitimate by not measuring it.

Trade-offs

Adjusting for difficulty requires a difficulty signal, and the obvious one — model confidence — is the same signal used for routing, which makes the adjustment partly circular. Independent signals such as inter-reviewer disagreement are better and more expensive to obtain.

The alternative is unadjusted metrics, which are simple and systematically wrong in a direction that gets worse as the system improves. A rough adjustment is substantially better than none, and the precision matters less than the acknowledgement.

Interview question

"Your model's automated share went from 60% to 80% this quarter and your reviewers' accuracy dropped. Explain what is most likely happening, and tell me what you would measure to confirm it."