Decoding & Generation
Greedy, beam, temperature, top-k, nucleus, and constrained generation into structured formats.
16concepts
143flashcards
119minutes of reading
- 01 Contrastive Decoding and DoLa If a small model's failures are an exaggerated version of a large model's failures, the difference between their logits is a usable quality signal, and the same trick works between the early and late layers of a single model.
- 02 Min-p and Typical Sampling Two later refinements to nucleus sampling, one that scales the truncation threshold to the top token's own confidence, one that truncates by information content rather than raw probability rank.
- 03 Minimum Bayes Risk Decoding Every standard decoder searches for the most probable sequence, and a decade of machine translation research shows that the mode of a neural sequence model is frequently degenerate; MBR replaces maximisation with expected-utility estimation.