Entropy-Adaptive Sampling

Two decode steps from the same model. In the first, the context is The capital of France is and the distribution puts 0.98 on one token. In the second, the context is the opening of a story and probability is spread across two hundred plausible continuations. Top-p at 0.9 truncates the first to a single token, which is right, and truncates the second to a pool wide enough to include tokens the model rates at a thousandth the probability of its favourite, which is where incoherence enters.

The parameter is fixed and the distribution is not. Adaptive samplers close that gap by deriving the truncation threshold from a statistic of the current distribution.

Three ways to make the threshold move

Min-p scales the cut to the top token's probability. Keep every token \(v\) with

A confident step with a 0.98 maximum and \(p_{\text{base}} = 0.1\) admits only tokens above 0.098, which is usually one token. A flat step with a 0.02 maximum admits everything above 0.002, a wide pool. One parameter, and the aggressiveness of truncation tracks the model's confidence automatically (Nguyen et al., ICLR 2025, arXiv:2407.01082).

Locally typical sampling takes an information-theoretic route. Human language, on this argument, sits near the expected information content rather than at the maximum: speakers do not choose the most predictable word at every turn, nor a wildly surprising one. Select the smallest set of tokens whose information content \(-\log p(v)\) is closest to the conditional entropy of the current distribution, and sample from that (Meister et al., TACL 2022, arXiv:2202.00666). This deliberately excludes the most probable token when the distribution is flat, which is the point.

Mirostat closes a feedback loop. It targets a specified perplexity for the generated text and adjusts the truncation at each step based on the surprisal observed so far, treating decoding as a control problem rather than a fixed rule (Basu et al., ICLR 2021, arXiv:2007.14966).

All three share one premise: the entropy of the next-token distribution says which regime you are in, and the sampler should respond to it.

Order of operations matters

Temperature and truncation do not commute, and the difference is larger than most people expect. Applying temperature first and then min-p changes which tokens survive, because temperature reshapes the ratio \(p(v) / \max_w p(w)\) that min-p thresholds on. Applying min-p first freezes the candidate set and then only redistributes mass inside it.

Different inference stacks make different choices, so a temperature=1.5, min_p=0.05 configuration ported between two servers can produce visibly different text. When reproducing a sampling result across engines, the sampler order is the first thing to check.

A live methodological dispute

Min-p was accepted as an oral at ICLR 2025 and adopted into Hugging Face Transformers and vLLM. It was then the subject of a detailed critique arguing the supporting evidence does not hold: human evaluation data from a third baseline was omitted without explanation, benchmark sweeps did not show min-p beating baselines once hyperparameter count was matched, and community-adoption figures cited in the paper were retracted from the camera-ready version (Schaeffer et al., 2025, Min-p, Max Exaggeration, arXiv:2506.13681).

The critique does not show min-p is harmful, and the sampler remains widely deployed. What it shows is that the field lacks an evaluation methodology able to separate decoders on open-ended generation, which is why decoding papers keep producing contested results. Treat any claimed sampler ranking on creative text as provisional; treat rankings on tasks with an exact answer as measurable.

When it breaks

Entropy is not calibrated confidence. A model can be uniformly wrong at low entropy. Adaptive samplers respond to distribution shape, not correctness, so a confidently hallucinating model gets a narrow pool that locks the hallucination in.

Repetition loops have low entropy by construction. Once a model enters a repeating cycle, each repeated token is highly predictable, so every entropy-adaptive rule narrows the pool and reinforces the loop. Adaptive truncation does not solve degeneration; it needs a repetition penalty alongside it.

Reasoning models change the workload. Long chain-of-thought traces spend most of their tokens on mechanical steps where near-greedy behaviour is correct, punctuated by a few high-entropy branch points. Uniform sampling parameters are badly matched to that profile, and whether the branch points can be detected reliably enough to sample them differently is unresolved.

Comparisons are rarely budget-matched. A sampler with two parameters tuned per task will beat one with a single fixed parameter, and much of the published gap between samplers is tuning effort rather than method.