Designing for Probabilistic Output
Why interfaces built on the assumption of correct output fail when output is usually correct, the design moves that make errors survivable, and the cost of hiding uncertainty.
Conventional software is deterministic and its failures are visible: something errors, something is empty, something crashes. A model's failure is a confident, well-formatted, plausible answer that is wrong, and an interface designed on the assumption of correctness gives the user nothing to work with when the assumption fails.
The design question is not how to prevent errors. It is how to make the user's experience acceptable at the error rate the system actually has.
The moves that work
Make verification cheaper than the task. If checking an output takes as long as doing the work, the system has saved nothing. Citations linking to the source passage, a diff against the previous version, a highlighted region of the input the output derives from: each turns verification from re-doing the work into checking a pointer. This is the highest-leverage move available and it is often architectural rather than visual.
Make correction cheap. Editable output beats regenerated output, because a user who can fix one wrong sentence keeps the value of the nine correct ones. Regeneration discards the good with the bad and costs another wait.
Show the work where it is load-bearing. Displaying which documents were retrieved, which tools were called, or which filters applied lets a user diagnose an odd answer instead of dismissing the system. It also reveals when the model is confidently answering from nothing.
Fail visibly rather than plausibly. A system that says it could not find relevant information is more useful than one that produces a confident answer from irrelevant context. Designing an "I don't know" path, and making the model take it, is a product decision that conflicts with the appearance of capability.
Design the low-confidence path deliberately. Not surfacing the answer at all, surfacing it with a caveat, routing to a human, or asking a clarifying question are four different responses to the same signal, and choosing per case beats a single global threshold.
When it breaks
Confidence displays are read as accuracy. A percentage next to an output is interpreted as a probability of correctness by users who have not seen the calibration curve, and models are frequently overconfident. Displaying an uncalibrated number is worse than displaying none, because it manufactures unwarranted trust with a number.
Streaming implies certainty. Text arriving token by token reads as thought in progress and as commitment, and there is no natural point at which a caveat can be attached to text already displayed. Buffering the first sentence to allow a check, or reserving a region for post-hoc annotations, are the available responses and both cost latency.
Undo does not exist for actions. An interface where the model acts, sends, books, deletes, needs an actual reversal path, and where none exists the confirmation is doing all the work. This is where the reversibility test from agent design meets interaction design.
Error rates users tolerate depend on stakes and on alternatives. A 5 percent error rate is fine for a draft and unacceptable for a medical summary, and the same rate is tolerable when the alternative is doing it manually and intolerable when the alternative is a reliable existing tool. The number alone does not determine whether the design works.
12 flashcards for this concept
Click a card to reveal the answer.