Guarantees Without Calibration: Conformal Prediction and the Limits of LLM Confidence

Ask a language model to rate its confidence and it will produce a number. Ask it on a hundred questions it answers wrongly a third of the time and the number will barely move. This is not a training defect that better data will fix; preference optimisation actively degrades the calibration that base models have, and the self-reported figure was never a probability in the first place. Yet the systems built on top of these models need to decide, per request, whether to answer, retrieve more, or hand off to a human.

Conformal prediction offers a trade that is unusual enough to be worth stating twice. It promises nothing about any individual prediction. It promises that, over the long run, the set of answers it returns will contain the correct one at least \(1-\alpha\) of the time, and that this holds in finite samples for any model, any score, and any data distribution, with one assumption: that your calibration data and your live traffic are exchangeable (Angelopoulos and Bates, 2021, A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification, arXiv:2107.07511). That single assumption is where all the engineering difficulty lives.

The most useful application to language models drops sets entirely. Instead of returning several candidate answers, a conformal back-off procedure makes the answer progressively less specific until the remaining claims can be guaranteed, reporting 80-90% correctness guarantees while retaining the majority of the model's original output on FActScore, NaturalQuestions and MATH (Mohri and Hashimoto, 2024, arXiv:2402.10978).

Why this matters: Every deployed LLM system already has an implicit uncertainty policy, usually a hard-coded threshold on a softmax value that nobody validated. Conformal prediction is the only widely available tool that turns that threshold into a statement with a proof attached, and knowing exactly what it does and does not promise is what separates a defensible safety argument from a number in a config file.

TL;DR

• Split conformal prediction converts any confidence score into prediction sets with a finite-sample coverage guarantee. The model can be arbitrarily miscalibrated; poor calibration costs you larger sets, not broken coverage.
• The guarantee is two-sided: coverage is at least \(1-\alpha\) and at most \(1-\alpha+1/(n+1)\), so with \(n = 500\) calibration examples the procedure overshoots the target by at most 0.2 percentage points.
• It is marginal, not conditional. A system can hit 90% coverage overall while covering 99% of easy queries and 40% of hard ones, which is precisely backwards from what a safety case needs.
• Exchangeability is violated by everything production does: traffic drift, model upgrades, users adapting, and reusing evaluation data that also guided prompt selection. Adaptive conformal inference recovers long-run coverage online without the assumption (Gibbs and Candès, 2021, arXiv:2106.00170).
• For generation, the construction moves off the label set: onto a calibrated stopping rule for sampling (Quach et al., ICLR 2024, arXiv:2306.10193) or onto how specific the claim is allowed to be.
• The score matters more than the theory. Sequence log-probability is length-biased and weak; semantic entropy, which clusters samples by meaning before measuring entropy, is a substantially better signal and cost 5-10 generations per query (Farquhar, Kossen, Kuhn and Gal, 2024, Nature 630(8017), 625-630).
• In production the payoff is usually abstention, not sets. A set of size one means answer, a large set means escalate, and the threshold should come from the cost of being wrong rather than from a round number.

At a Glance

flowchart LR
  M["Any model, any score"] --> C["Calibration set, n labelled examples"]
  C --> Q["Quantile q-hat of nonconformity scores"]
  Q --> S["Prediction set for a new input"]
  S --> D{"Set size"}
  D -->|"one"| A["Answer"]
  D -->|"many"| E["Abstain or escalate"]
  S --> W["Guarantee holds only under exchangeability"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class M,C blue
  class Q,S,D purple
  class A,E emerald
  class W amber

Everything on the left is free: no retraining, no calibration, no assumption about the model. Everything on the right depends on the amber box, and the amber box is what deployment breaks.

[IMAGE: Histogram of nonconformity scores from a calibration set with a vertical line at the 1-alpha quantile, and two example test inputs shown below with their scores mapped onto the same axis, one falling inside the threshold and one outside. Caption: "One number, computed once, defines every prediction set thereafter."]

Before Distribution-Free Guarantees

The idea that a classifier should be allowed to decline is older than deep learning. Chow's 1970 analysis of the optimal reject rule established the basic economics: given the cost of an error and the cost of a rejection, there is an optimal confidence threshold, and it is derived rather than chosen. That framing sat mostly unused during the era when accuracy was the only reported number.

Conformal prediction arrived as a general theory in the mid-2000s, developed by Vovk, Gammerman and Shafer, whose Algorithmic Learning in a Random World (Springer, 2005) established the finite-sample validity results the field still uses. It stayed a statistics-community tool for over a decade, partly because the transductive version was computationally impractical and partly because deep learning's own uncertainty literature was busy elsewhere.

That elsewhere produced the two baselines everyone still compares against: MC dropout, which reinterprets dropout at inference as approximate Bayesian inference (Gal and Ghahramani, 2015, arXiv:1506.02142), and deep ensembles, which simply train several models and measure disagreement, and which repeatedly outperformed more sophisticated Bayesian approximations (Lakshminarayanan et al., 2016, arXiv:1612.01474). Both need either architectural cooperation or several models, and neither survives contact with a 400-billion-parameter model behind an API.

Language models forced the question back open. The finding that large models are reasonably calibrated on multiple-choice formats, and that they can be asked to estimate the probability that their own answer is correct, made self-evaluation the default cheap signal (Kadavath et al., 2022, arXiv:2207.05221). The finding that entropy over token sequences confuses "many ways to say one thing" with "many things to say" produced semantic entropy, and its extension to confabulation detection reached Nature in 2024.

timeline
    title Uncertainty for models you cannot open
    1970 : Chow derives the optimal reject rule from error and rejection costs
    2005 : Vovk, Gammerman and Shafer establish conformal prediction's finite-sample validity
    2015 : MC dropout reinterprets dropout at test time as approximate Bayesian inference
         : Deep ensembles follow and outperform most Bayesian approximations
    2017 : Selective classification is formalised for deep networks with a target risk
    2021 : A practical introduction brings conformal prediction to machine learning practice
         : Adaptive conformal inference drops the exchangeability requirement online
    2023 : Semantic entropy clusters generations by meaning before measuring uncertainty
         : Conformal language modelling calibrates a stopping rule for sampling
    2024 : Conformal factuality trades specificity for a correctness guarantee
         : Semantic entropy for confabulation detection is published in Nature

[IMAGE: Diagram contrasting three eras of uncertainty tooling: Bayesian approximations requiring weight access, ensembles requiring multiple models, and conformal methods requiring only a held-out labelled set. Caption: "The move that mattered was giving up on the model's internals."]

How Conformal Prediction Actually Works

The construction, in four steps

Pick a nonconformity score \(s(x, y)\), large when label \(y\) looks wrong for input \(x\). For a classifier the standard choice is \(s = 1 - \hat{p}(y \mid x)\).

1. Hold out \(n\) labelled calibration examples, used for nothing else.
2. Compute \(s_i = s(x_i, y_i)\) for each.
3. Take \(\hat{q}\) as the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest of those scores.
4. For a new input \(x\), return \(C(x) = \{ y : s(x, y) \le \hat{q} \}\).

Then

The lower bound is the guarantee and the upper bound is what stops the procedure from being uselessly conservative. Both come from a single exchangeability argument: if the test point's score is exchangeable with the \(n\) calibration scores, its rank among them is uniform, and the quantile is exactly a statement about that rank.

Two consequences fall straight out of the arithmetic. The \(+1\) in \((n+1)\) is why the procedure needs \(n \ge 1/\alpha - 1\) calibration points to be non-trivial: at \(\alpha = 0.1\) you need at least 9, at \(\alpha = 0.01\) at least 99, and below that the required quantile exceeds the largest observed score and the set becomes everything. And the overshoot term \(1/(n+1)\) is why calibration sets in the hundreds are the practical floor: at \(n = 100\) you may over-cover by a full percentage point, at \(n = 1000\) by a tenth of one.

[IMAGE: Curve of average prediction-set size against the target error rate alpha for two scores of different quality, with the coverage guarantee shown as a flat line at 1 minus alpha for both. Caption: "Validity is identical for both scores. Everything you care about is the other axis."]

Choosing a score is the entire engineering problem

The theory is indifferent to the score. Your set sizes are not.

For multiple choice, the softmax over option tokens works directly, and the standard recipe applies unchanged, producing option sets with a coverage guarantee (Kumar et al., 2023, arXiv:2305.18404). Note the interaction with a known pathology: models carry a prior over the option ID tokens themselves, so a conformal set built on raw option probabilities inherits that bias, valid but wider than necessary.

For free-form generation there is no finite label set to threshold, so the construction moves. Conformal language modelling calibrates two rules at once: a stopping rule governing how many samples to draw into a candidate set, and a rejection rule that removes low-quality candidates, proving that the returned set contains at least one acceptable answer with high probability while remaining small in practice.

For long-form factual output, the most practical variant treats the answer as a set of claims and calibrates the level of specificity. Rather than returning five candidate biographies, the system returns one, with the unsupportable details removed. Its guarantee is over correctness of what remains, and the reported operating points retain most of the original output at 80-90% correctness.

Whatever the construction, the quality of the underlying signal determines set size, and this is where semantic entropy earns its cost. Sample several generations, cluster them by bidirectional entailment so two samples share a cluster only when each entails the other, sum probability mass per cluster, and take entropy over clusters. Low semantic entropy means the model keeps saying the same thing in different words. High means it keeps saying different things, which is the observable signature of a confabulation.

Adaptive conformal, for when exchangeability is a fiction

Production traffic drifts, so the guarantee decays and nothing reports it. The online repair is to treat \(\alpha\) as a controller state rather than a constant: after each observation, increase the effective \(\alpha\) when you have been over-covering and decrease it when you have been under-covering. This provably achieves the target coverage frequency over long intervals irrespective of the data-generating process, at the cost of the finite-sample exactness that the static version provides.

That trade is usually correct for deployment. A guarantee that is exact under an assumption you cannot verify is worth less than one that is asymptotic under conditions you actually face.

Seeing It in Motion

Calibration is a one-off pipeline; serving is a per-request one. Keeping them visually distinct is what stops teams from accidentally recalibrating on live traffic:

flowchart TB
  subgraph Offline["Calibration, run once per model version"]
    O1["Held-out labelled set"] --> O2["Score every example"]
    O2 --> O3["Sort scores"]
    O3 --> O4["q-hat at the corrected quantile"]
  end
  subgraph Online["Serving, per request"]
    S1["New input"] --> S2["Score every candidate"]
    S2 --> S3["Keep candidates under q-hat"]
    S3 --> S4["Set size decides the action"]
  end
  O4 --> S3
  S4 --> MON["Monitor realised coverage"]
  MON --> O1
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class O1,O2,O3,O4 blue
  class S1,S2,S3,S4 purple
  class MON amber

The request lifecycle, including the states most designs forget to define:

stateDiagram-v2
  [*] --> Scored
  Scored --> Singleton: set size is one
  Scored --> Small: set size two to k
  Scored --> Large: set size above k
  Singleton --> Answered
  Small --> Clarify: ask the user to disambiguate
  Large --> Escalate: hand to a human or a stronger model
  Clarify --> Scored: with the extra constraint
  Escalate --> [*]
  Answered --> [*]

[IMAGE: Risk-coverage curve with coverage on the x-axis and selective risk on the y-axis, three curves overlaid for softmax confidence, self-reported confidence, and semantic entropy, with the operating point implied by a given cost ratio marked on each. Caption: "The gate's quality is the whole curve; the threshold is one point on it."]

Watch It Run

By the Numbers

How much slack the guarantee carries, purely as arithmetic from \(1-\alpha \le \text{coverage} \le 1-\alpha+1/(n+1)\):

At \(\alpha = 0.01\) the minimum calibration size rises to 99, and at \(\alpha = 0.001\) to 999, which is the constraint that actually binds in high-stakes deployments: a 99.9% coverage guarantee requires at least a thousand labelled examples that are exchangeable with production traffic, and getting those is the project.

Reported results from the LLM-specific literature, for orientation rather than comparison:

Sources: Kumar et al., 2023, Quach et al., ICLR 2024, Mohri and Hashimoto, 2024, Farquhar et al., Nature 2024. The first table is exact arithmetic, not measurement. The second reports each paper's own claims on its own benchmarks; none has been replicated against the others.

A Concrete Example

A support-triage classifier routes tickets to one of twelve queues. The model returns a probability over queues. You want to be right at least 90% of the time about the queue being in the returned set, and you have 500 labelled tickets set aside that were used for nothing else.

Step 1, score the calibration set. For each ticket, \(s_i = 1 - \hat{p}(\text{true queue} \mid x_i)\). Scores range from 0.01 (confident and correct) to 0.99 (the true queue was ranked last).

Step 2, find the index. \(\lceil (500+1)(1-0.1) \rceil = \lceil 450.9 \rceil = 451\). Take the 451^st smallest score. Suppose it is \(\hat{q} = 0.83\).

Step 3, translate the threshold. Including every \(y\) with \(s(x,y) \le 0.83\) means including every queue whose predicted probability is at least \(0.17\). That is the entire rule, computed once.

Step 4, apply it to two live tickets. Ticket A has probabilities \(0.55, 0.30, 0.10, 0.05\) over its top four queues. Two clear the 0.17 bar, so the set is \(\{Q_1, Q_2\}\): the router cannot decide alone, and the sensible action is to present both to the agent. Ticket B has \(0.92, 0.05, 0.02, 0.01\). Only one clears the bar, so the set is a singleton and the ticket is auto-routed.

[IMAGE: Two horizontal probability bars for the worked example's tickets A and B over twelve queues, with a vertical cut line at 0.17 derived from q-hat = 0.83, showing ticket A producing a two-element set and ticket B a singleton. Caption: "One threshold, computed once from 500 labelled tickets, decides every routing action after it."]

Step 5, read the aggregate. Across a day of traffic, mean set size is 1.7 and 62% of tickets produce singletons. That 62% is your automation rate, and it is a direct consequence of the model's score quality, not of the conformal procedure. A better score at the same \(\alpha\) produces more singletons; the guarantee is unchanged.

Step 6, the model gets upgraded. A new checkpoint ships with different probability scaling. The old \(\hat{q} = 0.83\) is now applied to scores from a different distribution, exchangeability is broken, and measured coverage over the next week comes in at 0.82 against a promised 0.90. Nothing in the pipeline raised an error. This is the single most common way conformal guarantees fail in production, and the fix is procedural: recalibration is part of the deployment, not an afterthought.

Step 7, connect the threshold to money. Answering is worth it when \(p \cdot b - (1-p) \cdot c_{\text{wrong}} > -c_{\text{abstain}}\). With a benefit \(b = 1\) for correct auto-routing, a cost \(c_{\text{wrong}} = 8\) for a misroute (the ticket bounces, the customer waits a day), and \(c_{\text{abstain}} = 0.5\) for sending it to a human immediately, the break-even confidence is \(p > (c_{\text{wrong}} - c_{\text{abstain}}) / (b + c_{\text{wrong}}) = 7.5/9 = 0.83\). That number, not 0.9 and not 0.95, is where the gate belongs, and it changes the moment the cost of a misroute changes.

Where It Breaks

Marginal coverage is not the guarantee people hear

"90% coverage" is heard as "90% on every kind of query." It means 90% averaged over the input distribution. A system that covers 99% of routine queries and 40% of unusual ones satisfies it exactly, and the unusual ones are why you built the gate. Class-conditional and group-conditional variants restore per-group guarantees and require enough calibration data within each group, which is expensive precisely for the rare groups you care about.

Exchangeability is violated by ordinary operations

Model version bumps, prompt changes, seasonal traffic, a new customer segment, and users learning to phrase queries differently all break it. So does a subtler one: if the same held-out set informed prompt selection or few-shot choice, it is no longer exchangeable with test data, and the guarantee is void even though nothing looks different. Multi-turn and agentic settings are worse, because successive steps within one session are strongly dependent by construction.

[IMAGE: Time-series of realised coverage over twelve weeks against a flat promised line at 0.90, with a model version bump marked at week seven followed by a sustained drop to 0.82 and a recalibration event at week nine restoring the level. Caption: "Nothing errored. The only thing that changed was the guarantee."]

Validity without efficiency is theatre

A procedure that returns all twelve queues has perfect coverage and zero value. Efficiency, meaning average set size, is entirely inherited from the underlying score, and conformal prediction does nothing to improve it. If your softmax is uninformative, conformal prediction will tell you so, in the form of enormous sets, and that is genuinely useful diagnostic information rather than a failure of the method.

The guarantee is about labels, not truth

Coverage is defined against the labels in your calibration set. If those labels are noisy, ambiguous, or produced by an LLM judge with its own biases, the guarantee holds with respect to the labelling process, not reality. This is easy to lose sight of when the calibration set was itself generated to save annotation cost.

Scores from generation are structurally awkward

Sequence log-probability grows more negative with length and is dominated by lexical choices. Self-reported confidence is not stable in ordering across prompts. Sampling-based scores like semantic entropy are the strongest available option and cost 5-10 generations plus entailment checks per query, which relegates them to high-stakes paths or offline monitoring rather than every request.

Nothing here detects confident, consistent error

A model that has memorised a wrong association produces low entropy, high self-reported confidence, and a tight conformal set, and it is wrong. Every method in this article measures disagreement, spread, or rank; none measures truth. Detecting that failure requires an external check with independent access to the facts.

Alternative Designs

Conformal prediction is not a competitor to the scoring methods; it sits on top of them. The right way to read this table is that the first five rows produce scores, and the last two turn a score into a decision with a guarantee attached. Choosing a better score improves set size; choosing conformal calibration improves what you can claim.

How It Is Used in Practice

The deployments where this pays for itself share a shape: a high-volume decision, a measurable cost of error, and an existing human fallback. Clinical coding, insurance claim triage, content moderation queues, and document classification in regulated industries all fit, and in each the argument that convinces a reviewer is not "the model is accurate" but "the model handles the cases it can, at a measured error rate, and routes the rest."

Three operational details separate working systems from demonstrations. Calibration is versioned alongside the model, so a checkpoint change triggers recalibration rather than inheriting a stale quantile. Realised coverage is monitored continuously against the promised level, which doubles as the best available drift alarm: coverage falling below target is a distribution-shift signal with a precise meaning. And the abstention path is resourced, because a gate that escalates 38% of traffic to humans who do not exist is not a safety feature.

Where it does not fit: open-ended creative generation, where there is no label to cover; conversational assistants, where the exchangeability assumption dissolves across turns; and any setting where nobody will produce several hundred labelled examples from the live distribution. That last constraint eliminates more projects than the mathematics does.

Insights Worth Remembering

1. Calibration and coverage are different goods. You can have a badly calibrated model and a valid coverage guarantee at the same time. What miscalibration costs you is set size, which is the currency of usefulness, not validity.

2. The correction term is the whole proof. Taking the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest score rather than the plain empirical quantile is what makes the result exact in finite samples. It also sets the minimum calibration size at \(1/\alpha - 1\), which is why very tight guarantees are a data-collection problem.

3. Marginal coverage is the sharpest edge in this entire subject. The guarantee averages over exactly the distribution whose tails you were worried about. If your safety case needs per-group behaviour, you need per-group calibration data.

4. Exchangeability fails silently. There is no error, no exception, no metric that moves on its own. Realised-coverage monitoring is not optional instrumentation; it is the only thing standing between you and a guarantee that stopped being true a month ago.

5. The score is where the value is, and the wrapper is where the claim is. Improving from sequence probability to semantic entropy shrinks sets and raises automation rate. Adding conformal calibration does not improve the model at all; it lets you say something precise about it.

6. Uncertainty measures disagreement, not truth. Every technique here, from ensembles to entropy to conformal sets, is a consistency measurement. A confidently memorised falsehood is invisible to all of them.

7. The threshold belongs to the business, not to the statistics. The expected-value calculation produces the operating point. A gate set at 0.9 because 0.9 looks confident is an unpriced bet on the cost of being wrong.

Open Questions

Can conditional coverage be achieved at a bearable cost? Approximate conditional guarantees exist, and they consume calibration data per group. Whether there is a practical route to something close to conditional coverage for the long tail of query types in an open-domain system is unresolved.

What is the right exchangeability unit for agents? Steps within a trajectory are dependent, trajectories may be exchangeable, and it is not established what the correct calibration unit is or how much coverage degrades when the wrong one is chosen. This is measured for time series and largely unstudied for tool-using agents.

How much does semantic clustering cost in accuracy? Bidirectional-entailment clustering depends on a judge model, and its errors flow directly into the entropy estimate. The sensitivity of downstream decisions to that judge's quality has not been characterised across domains.

Does conformal back-off degrade usefulness more than sets do? Trading specificity for correctness is intuitively attractive and empirically retains most output on the benchmarks tested. Whether users prefer a vaguer guaranteed answer to a precise unguaranteed one is a human-factors question with no published evidence at scale.

Can guarantees survive model updates without full recalibration? Every deployment currently answers this by recalibrating. Whether a small amount of fresh data can be combined with a prior calibration set to maintain validity under a controlled model change is an active statistical question rather than a solved one.

Sources and Further Reading

1. Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. The founding treatment of conformal prediction's validity results.
2. Angelopoulos, A. N., & Bates, S. (2021). "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification." arXiv:2107.07511
3. Gibbs, I., & Candès, E. (2021). "Adaptive Conformal Inference Under Distribution Shift." NeurIPS 2021. arXiv:2106.00170
4. Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., & Barzilay, R. (2023). "Conformal Language Modeling." ICLR 2024. arXiv:2306.10193
5. Mohri, C., & Hashimoto, T. (2024). "Language Models with Conformal Factuality Guarantees." ICML 2024. arXiv:2402.10978
6. Kumar, B., et al. (2023). "Conformal Prediction with Large Language Models for Multi-Choice Question Answering." arXiv:2305.18404
7. Kuhn, L., Gal, Y., & Farquhar, S. (2023). "Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation." ICLR 2023. arXiv:2302.09664
8. Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). "Detecting hallucinations in large language models using semantic entropy." Nature, 630(8017), 625-630. doi:10.1038/s41586-024-07421-0
9. Kadavath, S., et al. (2022). "Language Models (Mostly) Know What They Know." arXiv:2207.05221
10. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016). "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles." NeurIPS 2017. arXiv:1612.01474
11. Gal, Y., & Ghahramani, Z. (2015). "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning." ICML 2016. arXiv:1506.02142
12. Geifman, Y., & El-Yaniv, R. (2017). "Selective Classification for Deep Neural Networks." NeurIPS 2017. arXiv:1705.08500
13. Chen, J., et al. (2023). "Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs." Findings of EMNLP 2023. arXiv:2310.11689
14. Chow, C. K. (1970). "On optimum recognition error and reject tradeoff." IEEE Transactions on Information Theory, 16(1), 41-46.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.