Information Theory for Language
Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression.
11concepts
61flashcards
85minutes of reading
- 01 Log-Probs and Token Probabilities What the log-probability numbers an API actually returns mean, how to turn them back into probabilities and perplexity, and the practical uses (reranking, confidence checks, hallucination flags) that make them worth reading instead of skipping.
- 02 N-Gram Models and Smoothing The pre-neural language model that ruled for three decades, why counting words breaks the moment you hit a sequence you have never seen, and the smoothing tricks invented to patch that hole.
- 03 Perplexity - Measuring a Language Model The oldest language-model metric, what "perplexity 12" actually means, and why you can never compare it across two models with different tokenisers.
- 04 Bits-per-Byte The tokenizer-invariant way to report a language model's compression performance, how to compute it from cross-entropy, and why serious scaling-law and benchmark papers report it instead of perplexity.
- 05 Cross-Entropy and KL Divergence The precise relationship between cross-entropy, entropy, and KL divergence, why minimising one is minimising "excess bits," and why forward and reverse KL pull a model in opposite directions.
- 06 Entropy and Surprise Why "surprising" has an exact numerical meaning, how -log p(x) turns probability into a unit of information, and what a language model's entropy floor tells you about a piece of text.
- 07 Maximum-Likelihood Estimation The estimation principle underneath every LLM's training objective, why "maximise probability of the data" and "minimise cross-entropy" are the exact same optimisation, and what changes when you add a prior.
- 08 Calibration of Language Models What it means for a model's stated confidence to be trustworthy, why pretrained next-token probabilities start out well calibrated and RLHF quietly breaks that, and how to actually measure the gap.
- 09 Language Modelling as Compression Every autoregressive language model is, exactly and not metaphorically, a lossless compressor; arithmetic coding is the bridge, and that equivalence is what let a text-only LLM out-compress PNG and FLAC on images and audio.
- 10 Mutual Information and Representations The information-theoretic quantity that measures how much one variable tells you about another, why it is the right lens for judging whether a learned representation actually captured something, and why estimating it in high dimensions is notoriously hard.
- 11 The Softmax Bottleneck Why a standard softmax output layer is a low-rank approximation to the true distribution of language, the "bottleneck" that caps what any single softmax can express, and the mixture trick that breaks it.