Decoding & Generation
Greedy, beam, temperature, top-k, nucleus, and constrained generation into structured formats.
16concepts
143flashcards
119minutes of reading
- 01 Beam Search Keeping the k best partial sequences alive instead of committing to one token at a time, and why the strategy that dominates machine translation actively hurts open-ended generation.
- 02 Determinism and Reproducibility in Decoding Why "temperature zero" and "same seed" both promise less determinism than they sound like they do, and where the real sources of nondeterminism live in an inference stack.
- 03 Entropy-Adaptive Sampling A fixed top-k or top-p threshold applies the same truncation to a distribution with one plausible continuation and to one with two hundred; adaptive samplers set the cut from the shape of the distribution itself.
- 04 Logit Bias and Masking Directly editing the logit vector before sampling, additive nudges for soft steering, hard -infinity masks for guaranteed exclusion, and why the two are not interchangeable.
- 05 Neural Text Degeneration The umbrella diagnosis behind most decoding research, why decoders that maximise sequence probability produce measurably worse text than decoders that sample from the model's actual distribution.
- 06 Nucleus (Top-p) Sampling Truncating by cumulative probability mass instead of a fixed rank, so the candidate pool automatically shrinks when the model is confident and grows when it is uncertain.
- 07 Repetition Penalty and No-Repeat N-Grams Two blunt but effective ways to stop a decoder from looping, subtracting from the logits of tokens already seen, versus hard-banning any n-gram that has already appeared.
- 08 Sampling and Decoding A language model outputs a probability distribution, not text; the decoding strategy turns that distribution into words and quietly decides whether the output is dull, unhinged, or right.
- 09 Stopping Criteria and EOS Calibration Generation ends when the model emits an end-of-sequence token, when a stop string matches, or when the token budget runs out, and the three failure modes look identical from outside while having completely different causes.
- 10 Structured Generation and Constrained Decoding How masking the logits at each decode step to only tokens a schema or grammar allows guarantees syntactically valid output, and where that guarantee stops.