Explanations and Their Effect on Reliance

On a beer-review sentiment task, crowd workers alone scored 0.82 accuracy and the model alone 0.84. Shown the model's recommendation with its confidence, the human-AI team reached 0.89, better than either. Adding a highlighted explanation of the model's reasoning gave 0.88, statistically indistinguishable, and across three tasks explanations never significantly beat the confidence-only baseline (Bansal et al., 2021, Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance, CHI, arXiv:2006.14779). Splitting the data by whether the model was right showed why. Explanations increased the chance that people accepted the recommendation regardless of its correctness: they helped when the model was right and, on two of the three tasks, hurt when it was wrong, so the net gain vanished.

The explainable-AI programme had assumed that seeing the reasons lets a person spot bad ones. That assumption is testable, and the neighbouring concepts on appropriate reliance and human-AI complementarity set out what success would look like. This concept is about the specific instrument of explanation, and why it so often fails that test.

What an explanation has to change

Take a binary decision where rejecting the AI means choosing the other answer. Let \(a\) be the AI's accuracy, \(r_c = P(\text{accept} \mid \text{AI correct})\) and \(r_w = P(\text{accept} \mid \text{AI wrong})\). Team accuracy is

An explanation is useful only if it widens the gap \(r_c - r_w\), the person's discrimination between good and bad recommendations. A cue that raises both rates is persuasion, not information.

With \(a = 0.84\), suppose a confidence-only interface yields \(r_c = 0.90\) and \(r_w = 0.40\), giving \(A = 0.756 + 0.096 = 0.852\). An explanation that lifts acceptance to \(r_c = 0.95\) but also \(r_w = 0.70\) gives \(A = 0.798 + 0.048 = 0.846\), worse, even though users now follow correct advice more often. These rates are illustrative, not Bansal et al.'s, but they show the shape of the finding: because correct recommendations dominate at high \(a\), a small gain on them is easily erased by a large loss on the rare wrong ones, and the rare wrong ones are where a human was supposed to add value.

Why explanations persuade

Buçinca, Malaya and Gajos read the failure through dual-process theory. People do not analyse each explanation; they form a general heuristic about whether the AI is competent, and the mere presence of an explanation reads as a signal of competence. Their response was cognitive forcing: designs that make heuristic acceptance harder, such as requiring an initial decision before revealing the AI's (update), showing the suggestion only on request (on demand), or imposing a delay (wait). In an experiment with 199 participants the forcing designs significantly reduced overreliance, measured as agreement with the AI when it was wrong, compared with plain explanation conditions (Buçinca, Malaya & Gajos, 2021, To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making, PACM HCI 5 CSCW1).

Two costs came with it. People rated the most effective designs least favourably, and trusted them less. The benefit also accrued mainly to participants high in Need for Cognition, so an intervention meant to protect users helped most those already inclined to think hard.

The disagreement: inherent failure or bad explanations?

Vasconcelos et al. challenged the pessimistic reading. Their account is strategic: a person weighs the cost of doing the task, the cost of verifying the AI through its explanation, and the payoff for being right. Prior studies, they argue, sat in a corner of that space where the explanation was nearly as costly to check as the task itself, so ignoring it was rational. Across five studies with 731 participants on a maze task, overreliance fell when the task was harder, when explanations were easier to understand and when correct answers paid more (Vasconcelos et al., 2023, Explanations Can Reduce Overreliance on AI Systems During Decision-Making, CSCW, arXiv:2212.06823).

The two accounts are not contradictory, but they point to different fixes. The dual-process view says change the interaction so people must think. The cost-benefit view says change the explanation so thinking is cheap, which connects directly to the verification-cost problem. A maze path can be checked at a glance; a SHAP bar chart over word features usually cannot, and that difference may be most of what separates the positive results from the null ones.

When it breaks

Proxy evaluations mislead. Asking people whether an explanation is helpful, or whether they can predict the model's output from it, is cheaper than measuring decision accuracy with the AI right and wrong. Subjective preference ran opposite to effectiveness in the forcing study, so these proxies can select the wrong design.

Explanation fidelity is a separate question. A post-hoc rationale that does not reflect the model's actual computation can be perfectly plausible, which makes it a stronger persuader and a worse discriminator.

Friction erodes in deployment. Laboratory participants tolerate an enforced wait; practitioners under throughput targets find workarounds, and a forcing function that is bypassed reduces nothing.

Aggregate accuracy hides the split. The cancellation Bansal et al. found is invisible unless results are reported separately for correct and incorrect AI recommendations. A deployment that tracks only overall accuracy cannot tell an explanation that informs from one that merely persuades.