Decoding & Generation
Greedy, beam, temperature, top-k, nucleus, and constrained generation into structured formats.
12concepts
51flashcards
89minutes of reading
- 01 Beam Search Keeping the k best partial sequences alive instead of committing to one token at a time, and why the strategy that dominates machine translation actively hurts open-ended generation.
- 02 Determinism and Reproducibility in Decoding Why "temperature zero" and "same seed" both promise less determinism than they sound like they do, and where the real sources of nondeterminism live in an inference stack.
- 03 Logit Bias and Masking Directly editing the logit vector before sampling, additive nudges for soft steering, hard -infinity masks for guaranteed exclusion, and why the two are not interchangeable.
- 04 Neural Text Degeneration The umbrella diagnosis behind most decoding research, why decoders that maximise sequence probability produce measurably worse text than decoders that sample from the model's actual distribution.
- 05 Nucleus (Top-p) Sampling Truncating by cumulative probability mass instead of a fixed rank, so the candidate pool automatically shrinks when the model is confident and grows when it is uncertain.
- 06 Repetition Penalty and No-Repeat N-Grams Two blunt but effective ways to stop a decoder from looping, subtracting from the logits of tokens already seen, versus hard-banning any n-gram that has already appeared.
- 07 Sampling and Decoding A language model outputs a probability distribution, not text; the decoding strategy turns that distribution into words and quietly decides whether the output is dull, unhinged, or right.
- 08 Structured Generation and Constrained Decoding How masking the logits at each decode step to only tokens a schema or grammar allows guarantees syntactically valid output, and where that guarantee stops.