Contrastive Decoding and DoLa

A 125-million-parameter language model repeats itself, loses the thread, and drifts off topic. A 13-billion-parameter model does the same things, less often. The observation that turned this into a decoding method: the failure modes are shared, so subtracting the small model's log-probabilities from the large model's cancels the shared pathology and leaves what the extra capacity actually bought.

Contrastive decoding uses that difference as the objective (Li et al., ACL 2023, arXiv:2210.15097). It needs no training, no reward model, and no reference text.

The objective and its guard rail

Call the large model the expert and the small one the amateur. The naive score is

Taken alone this is unusable, because the maximum of a log-ratio is typically some token the expert also considers absurd; it just happens to be one the amateur hates more. The fix is an explicit plausibility constraint that restricts the argmax to tokens the expert already endorses:

with \(\alpha\) commonly 0.1. Score only inside that set and assign minus infinity elsewhere. The constraint does the work that top-p does in nucleus sampling: it keeps the decoder inside the region the model considers reasonable, and the contrastive term picks within it.

Li and colleagues report that the method beat nucleus, top-k, and two other strong decoders in automatic and human evaluation across Wikipedia, news, and story domains. O'Brien and Lewis then applied it to reasoning rather than open-ended text and found LLaMA-65B with contrastive decoding outperforming LLaMA 2, GPT-3.5, and PaLM 2-L on HellaSwag, and beating LLaMA 2, GPT-3.5 and PaLM-540B on GSM8K, attributing the gain to suppression of specific abstract reasoning errors and of repetitive chain-of-thought loops (O'Brien & Lewis, 2023, arXiv:2309.09117).

DoLa contrasts the layers instead of the models

Loading a second model is an operational cost. DoLa removes it by observing that a transformer is already a stack of progressively better predictors. Project an intermediate layer's residual stream through the unembedding matrix and you get a distribution over the vocabulary; project the final layer and you get a better one. Factual knowledge in particular tends to be resolved in the later layers.

So contrast them. DoLa computes the mature-layer distribution, selects a premature layer dynamically as the one whose vocabulary distribution is furthest from the mature layer by Jensen-Shannon divergence, and takes the log-difference of the two (Chuang et al., ICLR 2024, arXiv:2309.03883). The reported effect is a 12 to 17 absolute point improvement on TruthfulQA for LLaMA-family models, at the cost of one extra unembedding projection per candidate layer.

The two methods are the same manoeuvre applied along different axes. Contrastive decoding contrasts across model scale; DoLa contrasts across depth. Both assume the weaker predictor's errors are a superset of the stronger one's.

When it breaks

The amateur is sometimes right. The assumption fails wherever small and large models agree for good reasons: copying a proper noun from the prompt, closing a bracket, following a strict output format. Contrastive decoding actively penalises tokens the amateur is confident about, so it degrades on exactly the mechanical parts of generation. Practitioners see this as broken JSON and mangled quotations.

Two models, two memory footprints. Even a 125M amateur adds weights, a second KV cache, and a second forward pass per step to the critical path. It is cheap relative to the expert and it is not free, and it complicates batching because the two models have different shapes.

Hyperparameter sensitivity. The plausibility threshold and the contrast strength interact with model pair, task, and prompt length. Reported gains come from tuned settings, and the same settings transferred to a different model family often do nothing.

Modern instruction-tuned models are already contrasted. Much of what contrastive decoding suppresses, repetition and topic drift, is what RLHF and long post-training already suppress. Gains reported on 2022-era base models do not transfer at full strength to a current instruction-tuned model, and this is the most common disappointment when the method is tried today.

DoLa's premature layer is a moving target. The dynamic selection depends on the divergence profile, which varies by token position and architecture. Models without a clean early-to-late knowledge gradient do not benefit, and there is no cheap way to know in advance which those are.