The Mode Is Not the Answer: How Decoding Stopped Being a Search Problem

In 2019 Felix Stahlberg and Bill Byrne did something nobody had bothered to do: they built a decoder that actually finds the optimum. Not a beam approximation, but exact inference, combining beam search with depth-first search to provably identify the highest-scoring translation under a trained Transformer on the WMT15 English-German test set. Then they looked at what it found. For more than 50% of the sentences, the model's globally best-scoring output was the empty string (Stahlberg & Byrne, EMNLP 2019, arXiv:1908.10090).

Beam search never finds that optimum. It cannot; the empty hypothesis is pruned early because its partial score at length one looks unremarkable. The system that was shipping translations to users worked because its search was bad, and the paper says so in as many words: vanilla neural machine translation depends on beam search errors to avoid producing nothing at all.

That is not a bug report about beam width. It is a statement that the objective was wrong. If the highest-probability sequence is degenerate, then every attempt to search harder is an attempt to fail more completely, and the entire framing of decoding as "find the best sequence" needs replacing.

Why this matters: Every generation call your system makes runs a decision rule, and the default rule optimises a quantity that only loosely tracks output quality. Minimum Bayes risk, contrastive decoding, and entropy-adaptive sampling are three different answers to the same question, and understanding what each is actually optimising is the difference between tuning parameters and choosing the right estimator.

TL;DR

• Exact search over a trained NMT model puts the global optimum on the empty string for over half of WMT15 English-German. Beam search's approximation error is load-bearing.
• Minimum Bayes risk decoding replaces argmax p(y|x) with maximum expected utility under the model's own distribution, estimated from samples. It has no equivalent of the beam search curse: quality does not degrade as the hypothesis space grows.
• MBR with a neural utility metric produced translations that humans preferred while scoring lower on model likelihood and lower on BLEU, which is the cleanest available separation of probability from quality.
• Contrastive decoding subtracts a small model's log-probabilities from a large model's, exploiting the fact that failure modes are shared and exaggerated in the smaller one. DoLa does the same trick between the early and late layers of one model, improving TruthfulQA by 12 to 17 absolute points.
• Self-consistency voting and best-of-N reranking are MBR in disguise, with exact-match and reward-model utilities respectively. The framework unifies techniques that were published as unrelated tricks.
• Naive MBR needs \(N^2\) utility evaluations and cannot stream a single token until every sample is complete, which rules it out of interactive products regardless of quality.
• The field cannot currently measure decoder quality on open-ended text well enough to settle disputes. An ICLR 2025 oral on min-p was followed by a detailed reanalysis arguing its evidence does not support its claims, and that argument was largely about methodology.

At a Glance

flowchart LR
    P["Prompt"] --> M["Model distribution"]
    M --> R{"Decision rule"}
    R -->|"argmax"| B["Beam search<br/>finds the mode"]
    R -->|"expected utility"| U["MBR<br/>samples then scores"]
    R -->|"log-ratio"| C["Contrastive<br/>expert minus amateur"]
    R -->|"truncate by shape"| S["Adaptive sampling<br/>min-p, typical"]
    B --> D["Short, generic,<br/>sometimes empty"]
    U --> Q["Consensus output"]
    C --> Q
    S --> Q

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class P,M blue
    class R slate
    class B,C,S,U purple
    class D rose
    class Q teal

The diagram compresses the argument. One model, one distribution, and four decision rules that read that distribution differently. Only the first is trying to find the mode, and it is the one that produces the pathology.

Before the Mode Was Suspect

Sequence-to-sequence models arrived with a decoder inherited from statistical machine translation, where beam search over a log-linear model with a language-model feature genuinely approximated the best translation. The inheritance came with an assumption nobody re-examined: that the highest-probability output under the model is the output you want.

The first cracks were empirical and were reported as tuning oddities. Koehn and Knowles listed beam search among six challenges for neural machine translation, alongside domain mismatch and rare words (Koehn & Knowles, WNMT 2017, arXiv:1706.03872). Practitioners found the same thing in their own systems: increasing beam width past about five made BLEU worse, not better, which is not how an approximation to a good objective is supposed to behave.

timeline
    title How the field stopped trusting the mode
    2017 : Koehn and Knowles list beam search among six NMT challenges
         : Practitioners find larger beams degrade quality
    2019 : Holtzman et al. name neural text degeneration and propose nucleus sampling
         : Stahlberg and Byrne show the empty string is the global optimum for over half of WMT15 En-De
    2020 : Eikema and Aziz argue the evidence indicts MAP decoding, not the model
         : Sampling-based MBR proposed as the alternative decision rule
    2021 : Freitag et al. pair MBR with BLEURT; humans prefer lower-likelihood outputs
         : Mirostat controls output perplexity with a feedback loop
    2022 : Contrastive decoding turns expert-minus-amateur into an objective
         : Locally typical sampling grounds truncation in information theory
    2023 : DoLa contrasts early and late layers instead of two models
         : Bertsch et al. show many modern techniques are MBR variants
    2024 : Min-p ties truncation to model confidence and ships in vLLM and Transformers
    2025 : A critical reanalysis disputes min-p's evidence and exposes the evaluation gap

Two 2019 papers reframed everything. Holtzman and colleagues characterised the failure on the generation side, showing that likelihood-maximising decoders produce text that is bland and strangely repetitive, and that the distributional statistics of machine text diverge visibly from human text; nucleus sampling was their response (Holtzman et al., ICLR 2020, arXiv:1904.09751). Stahlberg and Byrne supplied the mechanism on the translation side with the exact-search result.

Eikema and Aziz then drew the conclusion the field had been circling. The pathologies attributed to neural models and to maximum likelihood estimation were, on their analysis, mostly pathologies of the decision rule. Translation distributions reproduce various statistics of the data well; beam search strays from those statistics. The most likely translations accumulate so little probability mass that the mode can be considered essentially arbitrary. Their recommendation was to use decision rules that account for the distribution holistically, and they showed an approximation to MBR doing exactly that (Eikema & Aziz, COLING 2020, arXiv:2005.10283).

[IMAGE: Two probability-mass diagrams side by side. Left, a translation model's hypothesis space with a long flat plateau and a tiny isolated spike labelled "the mode, mass under 0.001". Right, the same space with a shaded region labelled "high-utility consensus region" covering the plateau. Caption: "The mode is a point estimate of a distribution that has no meaningful peak."]

How Decoding Stopped Being a Search Problem

The replacement framing is decision-theoretic and old. You are not searching for a string; you are choosing an action under uncertainty, and the right action minimises expected loss.

The decision rule

Fix a utility function \(u(h, y)\) that scores hypothesis \(h\) against a putative truth \(y\). The Bayes-optimal choice is the hypothesis maximising expected utility under your posterior over \(y\), which here is the model itself:

Set \(u\) to the indicator function \(\mathbb{1}[h = y]\) and this reduces exactly to MAP decoding. That is the important observation: beam search is MBR with a degenerate utility. It treats every wrong output as equally wrong, so it can only reward exact agreement, and in a space of \(10^{50}\) strings exact agreement is a useless criterion. Every other choice of utility gives a different, usually better, decision rule.

[IMAGE: A single axis of hypotheses with a curve for expected utility and a separate spike for model probability, the two maxima falling in different places, annotated "MAP picks here" and "MBR picks here". Caption: "Two objectives over the same hypothesis space, with maxima that need not coincide."]

Estimating the expectation

The expectation is over the full output space and is intractable. The estimator that made MBR practical draws \(N\) samples \(y_1 \dots y_N\) from the model, treats them as pseudo-references standing in for the unknown truth, and averages:

In the common form the candidate pool \(\mathcal{H}\) is the sample set itself, so the algorithm is: sample \(N\) outputs, build the \(N \times N\) utility matrix, return the row with the highest mean. The model is untouched. Only the selection changed.

Eikema and Aziz established the property that matters most for practice: this approximation has no equivalent of the beam search curse. Enlarging the hypothesis space helps, monotonically, which is the opposite of the behaviour that made beam width a tuning parameter with an optimum at five. They further showed that mode-seeking strategies are useful for constructing compact candidate sets even when they are bad at selecting from them, which is why hybrid schemes that generate with beam and select with MBR work well (Eikema & Aziz, EMNLP 2022, arXiv:2108.04718).

The utility function is the model of quality

With MBR, whatever you use as \(u\) becomes your definition of a good output, explicitly and inspectably. That is a design opportunity and the field took it.

Freitag and colleagues replaced surface-overlap utilities with BLEURT, a learned neural metric, and found significant improvement in human evaluation. The detail that makes the result important is what the winning translations looked like on the old metrics: lower model likelihood than beam output, and lower BLEU (Freitag et al., TACL 2022, arXiv:2111.09388). Three quantities that everyone had treated as proxies for each other came apart under a direct test, and human preference tracked the one that was chosen deliberately.

Contrast as a different route to the same place

Contrastive decoding arrives at non-mode-seeking behaviour by a different argument. Take a large model (the expert) and a small one (the amateur), and observe that the small model's failures are the large model's failures amplified: more repetition, more topic drift, more incoherence. So score tokens by the difference:

Unguarded, this is worse than useless, because the log-ratio is maximised by tokens the amateur finds nearly impossible, which are usually tokens the expert also rejects. The fix is a plausibility constraint that confines the choice to what the expert already endorses:

with \(\alpha\) typically 0.1. Li and colleagues reported this beating nucleus, top-k, and two other strong decoders in automatic and human evaluation across Wikipedia, news and story domains, with no training required (Li et al., ACL 2023, arXiv:2210.15097). O'Brien and Lewis then found it helps reasoning as well as open-ended text, with LLaMA-65B under contrastive decoding beating LLaMA 2, GPT-3.5 and PaLM 2-L on HellaSwag, and beating LLaMA 2, GPT-3.5 and PaLM-540B on GSM8K (O'Brien & Lewis, 2023, arXiv:2309.09117).

[IMAGE: Three stacked bar charts of the next-token distribution over the same eight-token shortlist: expert probabilities, amateur probabilities, and the resulting contrastive score, with tokens excluded by the plausibility constraint greyed out. Caption: "The plausibility constraint runs first; the log-ratio only chooses among survivors."]

DoLa removes the second model by contrasting along depth instead of scale. A transformer is already a stack of progressively better predictors; project an intermediate layer's residual stream through the unembedding and you get a worse distribution than the final layer's, with factual knowledge in particular concentrated late. DoLa picks the premature layer dynamically as the one furthest from the mature layer by Jensen-Shannon divergence and contrasts the two, improving TruthfulQA by 12 to 17 absolute points for LLaMA-family models (Chuang et al., ICLR 2024, arXiv:2309.03883).

The unification

Bertsch and colleagues made the connection explicit. Self-consistency voting is MBR where the utility is exact match on the extracted answer. Best-of-N reranking with a reward model is MBR where the utility ignores the pseudo-references entirely. Several other recent techniques fall out as MBR variants with different utilities and different candidate-generation strategies, which supplies a theoretical account for results that had been reported as independent empirical tricks (Bertsch et al., 2023, arXiv:2310.01387).

Once you see it, the shared failure mode is obvious too: all of them are optimising against a utility, and all of them Goodhart on it as \(N\) grows.

[IMAGE: A 2x2 grid of decision rules with axes "utility is exact match vs graded" and "candidates from beam vs sampling". Cells labelled MAP/beam search, self-consistency, MBR with neural metric, best-of-N reranking. Caption: "Four familiar techniques, one decision-theoretic frame."]

Seeing It in Motion

Two views of the same machinery. The first compares what each rule does with the same candidate set; the second traces the MBR pipeline end to end so the cost structure is visible.

flowchart TB
    subgraph gen["Generate"]
        S1["Sample N candidates<br/>ancestral, T equals 1.0"]
    end
    subgraph rules["Select"]
        direction LR
        M1["MAP<br/>highest log p"]
        M2["MBR<br/>highest mean utility"]
        M3["Best-of-N<br/>highest reward score"]
    end
    subgraph out["Result"]
        O1["Short and generic"]
        O2["Consensus of the sample"]
        O3["Reward model's favourite"]
    end
    S1 --> M1 --> O1
    S1 --> M2 --> O2
    S1 --> M3 --> O3
    O2 -.->|"cost: N squared utility calls"| W1["Latency"]
    O3 -.->|"cost: N reward passes"| W1

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class S1 blue
    class M1,M2,M3 purple
    class O1 rose
    class O2,O3 teal
    class W1 amber

The dotted edges carry the point that the boxes hide. MBR and best-of-N buy their quality with inference compute, and they buy it in different currencies: MBR pays in utility evaluations that scale quadratically, best-of-N pays in reward-model forward passes that scale linearly.

sequenceDiagram
    participant C as Client
    participant S as Sampler
    participant U as Utility metric
    participant R as Selector
    C->>S: prompt
    Note over S: N ancestral samples,<br/>no truncation
    S->>U: candidate set of size N
    loop N times N ordered pairs
        U->>U: score candidate against pseudo-reference
    end
    U->>R: utility matrix
    R->>R: row means, then argmax
    R->>C: single output, no streaming
    Note over C,R: first token arrives only<br/>after every sample completes

The final note is the operational verdict. There is no partial output to stream, so time to first token equals time to last token plus the scoring pass. That is why MBR is standard in offline translation and batch evaluation pipelines and absent from chat products.

By the Numbers

Sources: Stahlberg & Byrne (2019), Freitag et al. (2022), Chuang et al. (2024), Li et al. (2023), O'Brien & Lewis (2023), Schaeffer et al. (2025). The pair count is \(N(N-1)\) and is arithmetic rather than a measurement. The min-p row records a live dispute rather than a settled result; see Where It Breaks.

A Concrete Example

One sentence to summarise, five samples, and enough arithmetic to replay on paper.

Input. "The board approved the merger on Tuesday after a three-hour session."

Step 1: sample. Five ancestral samples at temperature 1.0, with their model log-probabilities:

Step 2: what MAP does. The highest log-probability candidate is D, "Approved.", at -2.1. This is the empty-string pathology in miniature: the shortest output accumulates the fewest sub-one factors and wins on probability while conveying nothing. A beam decoder does not literally return D, but it is biased in exactly this direction, which is why length-normalised beam scoring exists as a patch.

Step 3: choose a utility. Unigram F1 over content tokens, with multiset overlap:

Step 4: build the utility matrix. Ten unique pairs. Working the first one: A has 6 tokens, B has 11, and their multiset intersection is the twice plus board, approved, merger, tuesday, so 6 tokens overlap and \(u = 12/17 = 0.706\).

Step 5: row means.

• A: (0.706 + 0.588 + 0.286 + 0.706) / 4 = 0.572
• B: (0.706 + 0.727 + 0.167 + 0.909) / 4 = 0.627
• C: (0.588 + 0.727 + 0.167 + 0.818) / 4 = 0.575
• D: (0.286 + 0.167 + 0.167 + 0.167) / 4 = 0.197
• E: (0.706 + 0.909 + 0.818 + 0.167) / 4 = 0.650

Step 6: read the result. MBR selects E, at log-probability -6.9, the second least likely candidate in the set. MAP selects D, the most likely, at -2.1. The gap between the rules is 4.8 nats of model probability, and the direction of the disagreement is the whole point: the consensus estimate and the mode point at opposite ends of the candidate list.

D collapses under MBR for a structural reason worth naming. Its mean utility of 0.197 is low because it shares almost nothing with any other sample, and MBR punishes exactly that. Under MAP, brevity is rewarded; under MBR, brevity is isolation.

Note the tie-break too. If B and E had scored equally, the implementation falls back to model probability, which is how a mode-seeking criterion survives inside a non-mode-seeking rule as a secondary sort.

[IMAGE: The 5x5 utility matrix rendered as a heatmap, blue for low and teal for high, with row D a solid pale band and the E row visibly darkest. Right margin shows a bar chart of row means with a dashed line at the MAP choice. Caption: "MBR selects the row with the highest mean, not the sequence with the highest probability."]

Where It Breaks

The cost is quadratic and it is not hidden anywhere

Ten pairs at \(N=5\) is trivial. At \(N=100\) it is 9,900 ordered pairs, and if the utility is BLEURT or COMET, each pair is a neural forward pass. That is two orders of magnitude more inference than the generation itself. Confidence-based pruning and hyperparameter-free approximations recover much of it (Jinnai & Ariu, 2024, arXiv:2401.02749), but the shape of the tradeoff does not change: MBR converts inference compute into quality, and it needs a lot of it.

There is no first token to stream

Everything else in this article can stream. MBR cannot, by construction, because the selection depends on samples that do not exist yet. A product with a 300 ms time-to-first-token budget cannot use it at any \(N\). This is a harder constraint than cost, because cost can be bought and latency cannot.

Goodhart scales with N

Selecting from \(N\) samples by a learned metric is a search for that metric's blind spots, and the search gets more thorough as \(N\) grows. Measured utility climbs; true quality peaks and then declines. This is the same curve that limits best-of-N against a reward model, and it means "more samples is better" is true only up to a point that depends on how robust your utility is.

[IMAGE: Two curves against sample count N on a log axis. One labelled "measured utility" rises monotonically; one labelled "true quality" rises, peaks, and declines, with the crossover marked and annotated "depends on utility robustness, not on N". Caption: "The Goodhart curve is the shared failure mode of every sample-and-select decoder."]

Consensus selects for blandness on open-ended tasks

MBR returns the sample most similar to the others. When outputs are constrained, as in translation, the consensus is close to correct. When the task is genuinely open-ended, the sample set is legitimately diverse and its centroid is its least distinctive member: the most predictable story opening, the safest phrasing. The rule works as specified and the specification is wrong for that task.

Contrastive methods penalise the mechanical parts of generation

Contrastive decoding assumes the amateur's errors are a superset of the expert's. Where the two agree for good reasons, copying a name from the prompt, closing a bracket, obeying a JSON schema, the method actively penalises tokens the amateur is confident about. The visible symptom is malformed structured output, and it is why contrastive decoding and constrained decoding are awkward to combine.

Contrastive decoding also underdelivers on current models for an unglamorous reason: much of what it suppresses is what long post-training already suppresses. The gains reported on 2022-era base models overlap heavily with gains a 2026 instruction-tuned model already has.

The field cannot currently settle decoder disputes on open text

Min-p was accepted as an ICLR 2025 oral. A subsequent reanalysis argued the evidence does not support the claims: human-evaluation scores for a third baseline sampler, about a third of the collected data, were excluded from the methodology and results without explanation; benchmark sweeps showed no advantage once hyperparameter count was matched; and community-adoption figures were retracted from the camera-ready version (Schaeffer et al., 2025, arXiv:2506.13681).

Whatever the merits of min-p specifically, the episode is diagnostic. Ranking decoders on open-ended generation requires either large human studies or an LLM judge, and both have enough degrees of freedom to produce whichever ordering the analyst expects. Decoder claims on tasks with an exact answer are checkable. Claims about creative text should be read as provisional, whoever makes them.

Sampling temperature interacts with everything

MBR's theory calls for unbiased samples from the model, which means ancestral sampling at temperature 1.0. Practitioners routinely feed it nucleus samples, which biases the estimator toward the model's high-probability region and partially reintroduces the mode-seeking behaviour MBR exists to avoid. It still works. It is no longer the estimator the analysis describes, and the gap widens as truncation gets more aggressive.

Alternative Designs

The table has a diagonal to it. Moving down, quality rises, compute rises, and the ability to stream disappears. There is no row that dominates.

How It Is Used in Practice

MBR is standard in machine translation research and in the systems built from it, where outputs are short, offline batching is normal, and a strong neural metric already exists as an evaluation tool that can be repurposed as a utility. It also underlies competitive submissions to WMT-style shared tasks, where the compute budget is measured per test set rather than per user.

In LLM products it appears mostly in disguise. Self-consistency is deployed constantly under that name, and it is MBR with an exact-match utility on the extracted answer, applicable wherever answers are comparable. Best-of-N against a reward model or a verifier is deployed constantly under that name, and it is MBR with a reference-free utility. The engineers running them are usually not thinking in decision-theoretic terms, which costs them: the framing predicts the Goodhart curve, predicts that candidate diversity is the binding constraint rather than model quality, and predicts that the utility choice matters more than the sample count.

Contrastive methods have a narrower deployment story. DoLa is the more practical of the pair because it needs one model, and it shows up in factuality-sensitive pipelines. Full contrastive decoding remains mostly a research tool, partly because two resident models complicate serving and partly because the gains shrink on the models people actually deploy.

Entropy-adaptive samplers are the most widely shipped and the least well-evidenced. Min-p is in vLLM and Hugging Face Transformers, typical sampling is in most local-inference stacks, and the community consensus about which settings are good is built almost entirely on informal side-by-side comparisons.

[IMAGE: A deployment map with three lanes labelled offline batch, agentic and reasoning, interactive chat, and techniques placed in each: MBR and best-of-N in the first two, adaptive sampling and beam in the third, with a vertical axis showing latency budget from minutes to milliseconds. Caption: "Latency budget, not quality, decides which decision rule is available."]

[IMAGE: A cost breakdown for one request under four rules: beam search, nucleus sampling, best-of-N with N equals 16, and MBR with N equals 16, split into generation passes, scoring passes, and time to first token, drawn to scale. Caption: "MBR's scoring pass dominates, and its time to first token equals its total latency."]

Insights Worth Remembering

1. Beam search is MBR with an indicator utility. Setting \(u(h,y) = \mathbb{1}[h=y]\) recovers MAP decoding exactly. Every other decoder in this article is the same framework with a utility that gives partial credit, which is why they behave better in a space where exact agreement is meaningless.

2. The mode of a sequence model carries almost no probability mass. In a space of \(10^{50}\) strings, the single most likely sequence may hold less than a thousandth of the mass. Optimising a point estimate of a distribution with no peak is the root error, and length normalisation is a patch on the symptom.

3. Model probability, automatic metrics, and human preference are three different quantities. MBR with BLEURT produced outputs that scored lower on the first two and higher on the third. Any pipeline that uses one as a proxy for another should know the size of the gap it is assuming.

4. The failure mode of every sample-and-select method is the same, and it scales with N. More samples means a more thorough search for the utility's blind spots. Measured quality and true quality diverge at large \(N\), and the crossover point is a property of your utility, not of the method.

5. Streaming is a harder constraint than compute. Cost can be bought with hardware. A decision rule that needs every sample before it can emit a token cannot be made interactive at any price, which is why the highest-quality decoders live offline.

6. Contrastive methods trade mechanical reliability for semantic quality. Penalising what a small model is confident about penalises brackets, names, and schemas along with clichés. If your output is parsed by a machine, this is a real cost and not a rounding error.

7. A decoding claim about open-ended text is currently unfalsifiable in practice. The min-p episode was a dispute about human-evaluation methodology, baseline selection, and statistical treatment, not about the sampler's mathematics. Until the evaluation problem is solved, prefer decoders whose benefit is measurable on tasks with an answer.

Open Questions

Can MBR be made incremental? Every practical speedup so far prunes candidates, which reduces the constant and leaves the structure intact: sample everything, then select. Whether a variant exists that emits a stable prefix before all samples complete, perhaps by committing to tokens where all surviving candidates agree, is open. Nothing published resolves it.

What is the right utility for reasoning traces? For final answers, exact match works and self-consistency exploits it. For the trace itself, no good utility exists: two correct derivations by different routes should both score well, and a similarity metric penalises the diversity that made the sampling worthwhile. This is the main obstacle to applying MBR to reasoning rather than to answers.

Does contrastive decoding survive scale? The measured gains come from base models and from a specific era of instruction tuning. Whether the expert-minus-amateur signal remains informative when the expert has been through heavy post-training is not established. The mechanism suggests it should weaken, and no systematic study across current model families has been published.

How much of a decoder's reported gain is hyperparameter budget? The min-p reanalysis found no advantage once hyperparameter count was matched. That control is absent from most decoding papers, and applying it retroactively to the last five years of results would likely reorder them. This is a measurable question that nobody has funded.

Can the utility be learned per task without Goodharting? MBR makes the definition of quality an explicit, swappable component, which invites training it. But a learned utility optimised against is a reward model optimised against, with the same failure curve. Whether adversarial training, ensembling, or uncertainty-aware utilities push the crossover point far enough to matter is an open empirical question, and the honest answer today is that nobody knows where the ceiling is.

Sources and Further Reading

1. Stahlberg, F., & Byrne, B. (2019). "On NMT Search Errors and Model Errors: Cat Got Your Tongue?" EMNLP 2019. arXiv:1908.10090
2. Eikema, B., & Aziz, W. (2020). "Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation." COLING 2020. arXiv:2005.10283
3. Eikema, B., & Aziz, W. (2022). "Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation." EMNLP 2022. arXiv:2108.04718
4. Freitag, M., Grangier, D., Tan, Q., & Liang, B. (2022). "High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics." TACL. arXiv:2111.09388
5. Bertsch, A., Xie, A., Neubig, G., & Gormley, M. R. (2023). "It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk." arXiv:2310.01387
6. Jinnai, Y., & Ariu, K. (2024). "Hyperparameter-Free Approach for Faster Minimum Bayes-Risk Decoding." arXiv:2401.02749
7. Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., & Lewis, M. (2023). "Contrastive Decoding: Open-ended Text Generation as Optimization." ACL 2023. arXiv:2210.15097
8. O'Brien, S., & Lewis, M. (2023). "Contrastive Decoding Improves Reasoning in Large Language Models." arXiv:2309.09117
9. Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2024). "DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models." ICLR 2024. arXiv:2309.03883
10. Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). "The Curious Case of Neural Text Degeneration." ICLR 2020. arXiv:1904.09751
11. Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2022). "Locally Typical Sampling." TACL. arXiv:2202.00666
12. Basu, S., Ramachandran, G. S., Keskar, N. S., & Varshney, L. R. (2021). "Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity." ICLR 2021. arXiv:2007.14966
13. Nguyen, M. N., Baker, A., Neo, C., Roush, A., Kirsch, A., & Shwartz-Ziv, R. (2025). "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs." ICLR 2025. arXiv:2407.01082
14. Schaeffer, R., Kazdan, J., & Denisov-Blanch, Y. (2025). "Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models." arXiv:2506.13681
15. Koehn, P., & Knowles, R. (2017). "Six Challenges for Neural Machine Translation." First Workshop on Neural Machine Translation. arXiv:1706.03872
16. Tam, Z. R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., & Chen, Y.-N. (2024). "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models." arXiv:2408.02442

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.