Decoding & Generation
Greedy, beam, temperature, top-k, nucleus, and constrained generation into structured formats.
12concepts
51flashcards
89minutes of reading
- 01 Greedy Decoding The simplest possible decoder, take the single most probable token at every step, and why that local optimality guarantees nothing about the sentence you end up with.
- 02 Temperature Sampling Dividing logits by a scalar before softmax to sharpen or flatten a model's output distribution, and why temperature alone is a poor substitute for a real safety mechanism.
- 03 Top-k Sampling Truncating the vocabulary to a fixed-size shortlist of the k most probable tokens before sampling, and why a fixed k is systematically the wrong size for at least some fraction of every distribution.
- 04 Beam Search Keeping the k best partial sequences alive instead of committing to one token at a time, and why the strategy that dominates machine translation actively hurts open-ended generation.
- 05 Determinism and Reproducibility in Decoding Why "temperature zero" and "same seed" both promise less determinism than they sound like they do, and where the real sources of nondeterminism live in an inference stack.
- 06 Logit Bias and Masking Directly editing the logit vector before sampling, additive nudges for soft steering, hard -infinity masks for guaranteed exclusion, and why the two are not interchangeable.
- 07 Neural Text Degeneration The umbrella diagnosis behind most decoding research, why decoders that maximise sequence probability produce measurably worse text than decoders that sample from the model's actual distribution.
- 08 Nucleus (Top-p) Sampling Truncating by cumulative probability mass instead of a fixed rank, so the candidate pool automatically shrinks when the model is confident and grows when it is uncertain.
- 09 Repetition Penalty and No-Repeat N-Grams Two blunt but effective ways to stop a decoder from looping, subtracting from the logits of tokens already seen, versus hard-banning any n-gram that has already appeared.
- 10 Sampling and Decoding A language model outputs a probability distribution, not text; the decoding strategy turns that distribution into words and quietly decides whether the output is dull, unhinged, or right.
- 11 Structured Generation and Constrained Decoding How masking the logits at each decode step to only tokens a schema or grammar allows guarantees syntactically valid output, and where that guarantee stops.
- 12 Min-p and Typical Sampling Two later refinements to nucleus sampling, one that scales the truncation threshold to the top token's own confidence, one that truncates by information content rather than raw probability rank.