Information Theory for Language
Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression.
16concepts
161flashcards
126minutes of reading
- 01 Bits-per-Byte The tokenizer-invariant way to report a language model's compression performance, how to compute it from cross-entropy, and why serious scaling-law and benchmark papers report it instead of perplexity.
- 02 Cross-Entropy and KL Divergence The precise relationship between cross-entropy, entropy, and KL divergence, why minimising one is minimising "excess bits," and why forward and reverse KL pull a model in opposite directions.
- 03 Entropy and Surprise Why "surprising" has an exact numerical meaning, how -log p(x) turns probability into a unit of information, and what a language model's entropy floor tells you about a piece of text.
- 04 Maximum-Likelihood Estimation The estimation principle underneath every LLM's training objective, why "maximise probability of the data" and "minimise cross-entropy" are the exact same optimisation, and what changes when you add a prior.