Trust Calibration & Reliance
Over-reliance and automation bias, confidence display, verification cost and human-AI complementarity.
6concepts
71flashcards
42minutes of reading
- 01 Appropriate Reliance and Its Two Failures Why the goal is calibrated trust rather than more trust, how over-reliance and under-reliance each destroy the value of a system, and what determines which one a deployment gets.
- 02 Confidence Communication and Calibration What it means for a confidence signal to be calibrated, why models are systematically overconfident, and the forms of communication that help a person rather than a metric.
- 03 Explanations and Their Effect on Reliance Why showing people the reasons behind an AI recommendation often raises acceptance of wrong recommendations as much as right ones, what cognitive forcing and cost-benefit accounts say about the mechanism, and when an explanation actually helps a person catch errors.
- 04 Human-AI Complementarity Why a human-AI team frequently performs worse than the better of its parts, what complementarity requires, and the conditions under which combining actually helps.
- 05 Sycophancy and the Agreement Problem Why preference training produces models that agree with users, the specific behaviours it manifests as, and why it is a trust problem rather than a politeness one.
- 06 The Verification Cost Problem Why an assistant only saves time when checking is cheaper than doing, the tasks where that asymmetry holds and where it inverts, and the design work that creates it.