Deciding Where the Human Stays
The four automation levels available for any decision, the expected-cost calculation that selects between them, and why partial automation is usually right and usually hardest to design.
"Should a human review this?" is treated as an ethical question and is mostly an economic one with an ethical constraint. The calculation is tractable, the inputs are estimable, and doing it explicitly produces better designs than either automating everything or reviewing everything.
The four levels
Human does the work, AI assists. The model surfaces information, drafts, or suggests, and the person decides and acts. Lowest risk, smallest efficiency gain, and appropriate where errors are costly and volume is low.
AI does the work, human reviews all of it. Every output is checked before it takes effect. The gain is real, since reviewing is faster than producing, and it is bounded by review capacity, which becomes the throughput ceiling.
AI acts, human reviews a sample or the uncertain cases. Selective review by confidence, by stakes, or by a random sample. This is where most well-designed systems land, and it is the hardest to build because it requires a reliable signal for what to route.
AI acts autonomously. Appropriate where errors are cheap, reversible and detectable, and where volume makes review impossible. Spam filtering and recommendation ranking live here.
The calculation
For a decision with automation rate \(a\), error rate \(e\), cost per error \(C_e\), cost per human review \(C_r\) and volume \(V\):
Automating is worth it when \(e \cdot C_e < C_r\), which is to say when the expected cost of an error is less than the cost of a review. This makes the tradeoff explicit and shows what changes it: reducing the error rate, reducing the cost of an error through reversibility, or reducing review cost through better interfaces all move the threshold.
The ethical constraint sits on top: some decisions require human involvement regardless of the arithmetic, because of regulation, because the affected person is entitled to it, or because the harm is not the kind that averages. Those are excluded from the calculation rather than priced into it.
When it breaks
Review capacity is the real constraint. Full review makes the human the bottleneck, so throughput is capped by reviewers rather than by the model. Systems designed for full review at pilot volume fail at production volume, and the failure looks like a queue rather than an error.
Reviewers under load stop reviewing. A person approving a hundred outputs an hour is not evaluating each one, and approval rate approaches one. Automation bias makes this worse: people accept a machine's suggestion more readily than they would evaluate the same claim unaided. Review that is not resourced to be real is a control on paper only.
Selective review needs a signal that works. Routing by model confidence assumes calibration, and an overconfident model routes exactly the wrong cases. Where calibration is poor, random sampling plus rule-based routing on stakes is more honest than a confidence threshold that looks principled and is not.
The human needs enough context to add value. A reviewer shown an output without the input, the retrieved evidence and the alternatives cannot do better than accept. Designing the review interface is most of designing the control, and a review step without it is a delay rather than a safeguard.
12 flashcards for this concept
Click a card to reveal the answer.