Information Theory for Language
Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression.
11concepts
61flashcards
85minutes of reading
- 01 Log-Probs and Token Probabilities What the log-probability numbers an API actually returns mean, how to turn them back into probabilities and perplexity, and the practical uses (reranking, confidence checks, hallucination flags) that make them worth reading instead of skipping.
- 02 N-Gram Models and Smoothing The pre-neural language model that ruled for three decades, why counting words breaks the moment you hit a sequence you have never seen, and the smoothing tricks invented to patch that hole.
- 03 Perplexity - Measuring a Language Model The oldest language-model metric, what "perplexity 12" actually means, and why you can never compare it across two models with different tokenisers.