Confidence Communication and Calibration
What it means for a confidence signal to be calibrated, why models are systematically overconfident, and the forms of communication that help a person rather than a metric.
A confidence signal is useful exactly to the extent that it is calibrated: among all outputs presented at 80 percent confidence, 80 percent should be correct. If that does not hold, the signal is worse than nothing, because it replaces a user's vague uncertainty with a specific and wrong belief.
Measuring calibration
The reliability diagram is the primary instrument: bin predictions by stated confidence, plot the observed accuracy per bin against the stated confidence, and compare against the diagonal. Expected calibration error summarises the gap as a weighted average of per-bin discrepancies, and it hides direction and shape, so the diagram should be read alongside it.
Modern neural networks are systematically overconfident, and the effect grew with capacity and with the training practices that improve accuracy. Temperature scaling, fitting a single scalar on a validation set to divide the logits, corrects much of it for classifiers at negligible cost and is close to free to implement.
For language models the picture is more complicated. Token-level probabilities are one signal and answer-level correctness is another, so the useful quantity is usually the probability assigned to the full answer or a self-reported confidence, and both need calibrating against observed correctness rather than assumed. Models asked to state their confidence in words are typically overconfident in a way that persists after instruction to be cautious.
Communicating it
Numeric probability is precise, and it is misread by most people, who treat "70 percent" as "probably right" without the frequency interpretation that makes it actionable.
Verbal qualifiers map to wide and inconsistent probability ranges across individuals, so "likely" means different things to different readers. Pairing a verbal expression with a number, and using a fixed internal mapping, narrows the variance.
Categorical bands, high, medium and low, discard resolution and are robust to miscalibration within a band, which is often the right trade when calibration is imperfect.
Behavioural expression is frequently the best of the four: instead of stating confidence, change what the system does. Show two candidate answers when uncertain, ask a clarifying question, route to a human, or decline. This communicates uncertainty through the interaction rather than through a number the user has to interpret.
When it breaks
Calibration is distribution-specific. A model calibrated on the validation set is not calibrated on shifted production data, and the shift is invisible without ongoing measurement. Calibration belongs in monitoring, not only in evaluation.
Selective prediction changes the population. A system that abstains on its least confident cases has a remaining population whose calibration differs, so the confidence signal must be recalibrated on the population it actually serves.
Confidence and correctness diverge on out-of-distribution input. The case where a user most needs a warning is the one where the model has no basis for one, since it has not seen anything like the input. Out-of-distribution detection is a separate mechanism and is not obtainable from the confidence score.
Displayed confidence changes behaviour asymmetrically. Users act on high confidence more than they discount low confidence, so an overconfident signal does more damage than an underconfident one. Where calibration is uncertain, biasing the display toward caution is the asymmetric choice.
14 flashcards for this concept
Click a card to reveal the answer.