Information Theory for Language
Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression.
16concepts
161flashcards
126minutes of reading
- 01 Calibration of Language Models What it means for a model's stated confidence to be trustworthy, why pretrained next-token probabilities start out well calibrated and RLHF quietly breaks that, and how to actually measure the gap.
- 02 Channel Capacity and the Noisy Channel Shannon's capacity theorem, why it says reliable communication is possible at any rate below capacity and impossible above it, and why the noisy-channel decomposition keeps reappearing in language modelling.
- 03 Kolmogorov Complexity and MDL The shortest program that outputs a string is the ultimate measure of its information content, why it is uncomputable, and how minimum description length turns that uncomputable ideal into a usable model selection principle.
- 04 Language Modelling as Compression Every autoregressive language model is, exactly and not metaphorically, a lossless compressor; arithmetic coding is the bridge, and that equivalence is what let a text-only LLM out-compress PNG and FLAC on images and audio.
- 05 Mutual Information and Representations The information-theoretic quantity that measures how much one variable tells you about another, why it is the right lens for judging whether a learned representation actually captured something, and why estimating it in high dimensions is notoriously hard.
- 06 Rate-Distortion Theory The theory of how few bits a source needs when perfect reconstruction is not required, why every compression decision in an LLM stack is a point on one curve, and what the curve's shape tells you.
- 07 The Softmax Bottleneck Why a standard softmax output layer is a low-rank approximation to the true distribution of language, the "bottleneck" that caps what any single softmax can express, and the mixture trick that breaks it.
- 08 Typical Sets and the AEP Why almost all probability mass sits on a vanishingly small set of sequences that are individually unremarkable, and why the most likely sequence is usually not a typical one.
- 09 f-Divergences Beyond KL KL is one member of a family generated by a convex function, the choice of member decides whether your model covers the data or collapses onto a mode, and some tasks need a divergence that is not in the family at all.