Appropriate Reliance and Its Two Failures
Why the goal is calibrated trust rather than more trust, how over-reliance and under-reliance each destroy the value of a system, and what determines which one a deployment gets.
Product teams optimise for trust as though more is better. The target is not more trust; it is trust matched to reliability, so that a person accepts the system when it is right and rejects it when it is wrong. Both deviations from that destroy the value, and they call for opposite interventions.
Over-reliance
The user accepts outputs without the scrutiny the error rate warrants. It appears as automation bias, the documented tendency to accept a machine's suggestion more readily than one would evaluate the same claim unaided, and as complacency, the gradual withdrawal of attention from a system that has been right for a while.
Over-reliance is the more dangerous failure because it is invisible until an error propagates. Everything looks efficient: high acceptance, fast throughput, no complaints. The system's error rate has simply become the joint system's error rate, since the human is not filtering anything.
It arises from a system that is right most of the time, from fluent and confident presentation, from reviewer time pressure, and from a design that makes accepting cheaper than checking.
Under-reliance
The user rejects or ignores outputs that are correct. It appears as algorithm aversion, the well-documented tendency to abandon an algorithm after seeing it err, even when it outperforms the human alternative, and as a general refusal to use a tool whose failures are memorable.
Under-reliance wastes the investment and is easier to see: usage is low, override rates are high, people work around the system. It arises from an early bad experience, from opaque output that cannot be verified, from a mismatch between the presented confidence and the observed accuracy, and from a system that solves a problem the user did not have.
What moves reliance toward calibration
Accurate confidence signals, but only if calibrated, since an uncalibrated signal actively miscalibrates the user.
Verification affordances, which let a user check cheaply and so make appropriate scepticism affordable. Where checking is expensive, users either accept everything or reject everything, and both are miscalibrations produced by the interface.
Exposure to the error distribution. A user who has seen the system's characteristic failures develops a mental model of when to trust it. Hiding failures produces users with no model, who then swing between the two failures.
Feedback on their own reliance. A user who learns that they overrode ten correct outputs last month is being given the information calibration requires and that nothing else in the system provides.
When it breaks
Calibration is per user, per task and per session. An expert on the task calibrates differently from a novice, and the same person calibrates differently under time pressure. A single design cannot produce appropriate reliance across all of them.
Improving the model can worsen calibration. A system that becomes more accurate invites more reliance, and if the remaining errors are concentrated in a hard subgroup, users trained on the improved average will be exactly wrong where it matters.
Recovering from under-reliance is harder than avoiding it. Algorithm aversion is asymmetric: a single salient failure costs more trust than many successes rebuild. This is an argument for launching narrow, where the system is reliably good, rather than broad and occasionally embarrassing.
12 flashcards for this concept
Click a card to reveal the answer.