Information Theory for Language
Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression.
11concepts
61flashcards
85minutes of reading
- 01 Calibration of Language Models What it means for a model's stated confidence to be trustworthy, why pretrained next-token probabilities start out well calibrated and RLHF quietly breaks that, and how to actually measure the gap.
- 02 Language Modelling as Compression Every autoregressive language model is, exactly and not metaphorically, a lossless compressor; arithmetic coding is the bridge, and that equivalence is what let a text-only LLM out-compress PNG and FLAC on images and audio.
- 03 Mutual Information and Representations The information-theoretic quantity that measures how much one variable tells you about another, why it is the right lens for judging whether a learned representation actually captured something, and why estimating it in high dimensions is notoriously hard.
- 04 The Softmax Bottleneck Why a standard softmax output layer is a low-rank approximation to the true distribution of language, the "bottleneck" that caps what any single softmax can express, and the mixture trick that breaks it.