Decoding & Generation
Greedy, beam, temperature, top-k, nucleus, and constrained generation into structured formats.
16concepts
143flashcards
119minutes of reading
- 01 Greedy Decoding The simplest possible decoder, take the single most probable token at every step, and why that local optimality guarantees nothing about the sentence you end up with.
- 02 Temperature Sampling Dividing logits by a scalar before softmax to sharpen or flatten a model's output distribution, and why temperature alone is a poor substitute for a real safety mechanism.
- 03 Top-k Sampling Truncating the vocabulary to a fixed-size shortlist of the k most probable tokens before sampling, and why a fixed k is systematically the wrong size for at least some fraction of every distribution.
- 04 Beam Search Keeping the k best partial sequences alive instead of committing to one token at a time, and why the strategy that dominates machine translation actively hurts open-ended generation.
- 05 Determinism and Reproducibility in Decoding Why "temperature zero" and "same seed" both promise less determinism than they sound like they do, and where the real sources of nondeterminism live in an inference stack.
- 06 Entropy-Adaptive Sampling A fixed top-k or top-p threshold applies the same truncation to a distribution with one plausible continuation and to one with two hundred; adaptive samplers set the cut from the shape of the distribution itself.
- 07 Logit Bias and Masking Directly editing the logit vector before sampling, additive nudges for soft steering, hard -infinity masks for guaranteed exclusion, and why the two are not interchangeable.
- 08 Neural Text Degeneration The umbrella diagnosis behind most decoding research, why decoders that maximise sequence probability produce measurably worse text than decoders that sample from the model's actual distribution.
- 09 Nucleus (Top-p) Sampling Truncating by cumulative probability mass instead of a fixed rank, so the candidate pool automatically shrinks when the model is confident and grows when it is uncertain.
- 10 Repetition Penalty and No-Repeat N-Grams Two blunt but effective ways to stop a decoder from looping, subtracting from the logits of tokens already seen, versus hard-banning any n-gram that has already appeared.
- 11 Sampling and Decoding A language model outputs a probability distribution, not text; the decoding strategy turns that distribution into words and quietly decides whether the output is dull, unhinged, or right.
- 12 Stopping Criteria and EOS Calibration Generation ends when the model emits an end-of-sequence token, when a stop string matches, or when the token budget runs out, and the three failure modes look identical from outside while having completely different causes.
- 13 Structured Generation and Constrained Decoding How masking the logits at each decode step to only tokens a schema or grammar allows guarantees syntactically valid output, and where that guarantee stops.
- 14 Contrastive Decoding and DoLa If a small model's failures are an exaggerated version of a large model's failures, the difference between their logits is a usable quality signal, and the same trick works between the early and late layers of a single model.
- 15 Min-p and Typical Sampling Two later refinements to nucleus sampling, one that scales the truncation threshold to the top token's own confidence, one that truncates by information content rather than raw probability rank.
- 16 Minimum Bayes Risk Decoding Every standard decoder searches for the most probable sequence, and a decade of machine translation research shows that the mode of a neural sequence model is frequently degenerate; MBR replaces maximisation with expected-utility estimation.