Language Modelling as Compression

Here is a claim stated precisely rather than loosely: a probabilistic model p(x) and a lossless compressor are the same mathematical object. You can convert either one into the other with zero loss of performance. This is not a metaphor borrowed to make information theory feel relevant to deep learning; it is a direct consequence of Shannon's source coding theorem, and it is the reason a large language model trained purely on text was able to out-compress purpose-built image and audio codecs it had never seen data from.

Arithmetic coding, in one paragraph

Arithmetic coding is the mechanism that makes the equivalence constructive rather than theoretical. Start with the interval [0, 1). For each symbol in the sequence, narrow the current interval into sub-intervals sized in proportion to the model's predicted probability for each possible next symbol, then keep only the sub-interval matching the symbol that actually occurred. A symbol the model thought was likely gets a wide sub-interval, costing few bits to specify; a symbol the model thought unlikely gets a narrow one, costing many. After the whole sequence, transmit any number inside the final tiny interval, and that number, decoded by running the identical model forward step by step, exactly reconstructs the original sequence. The number of bits this takes, on average, converges to the model's cross-entropy on the data (see entropy-and-surprise). A better predictive model is, by this construction, a better compressor, with no additional machinery required.

The Deletang et al. experiment

Deletang et al., 2023 (arXiv:2309.10668) made this concrete in a way that got attention well outside the compression-research community. They took language models trained purely on text and plugged them, unmodified, into an arithmetic coder as the probability engine, then fed the coder raw bytes of data the model had never trained on: ImageNet image patches and LibriSpeech audio clips. The result: a general-purpose LLM, evaluated purely as the predictive engine inside an arithmetic coder, compressed these out-of-domain images and audio more tightly than PNG and FLAC, codecs built specifically for those data types. The mechanism is exactly the one above: compression ratio is a direct function of predictive accuracy, and a large enough general-purpose predictor, trained on enough varied structure, can out-predict a small hand-engineered domain-specific model, even completely off-distribution.

Why this matters beyond a benchmark

The result is often cited as evidence for a stronger claim: that prediction, compression, and something like "understanding" might be three names for the same underlying capacity. This is not a new idea, it motivates the Hutter Prize, a standing reward for compressing a fixed sample of Wikipedia text, on the explicit premise that better compression requires better modelling of the structure and semantics of the text, not just better bit-shuffling. It also connects directly to the minimum description length (MDL) principle: given a choice between models that fit data equally well, the one with the shortest total description (model plus compressed residual) is the one that has captured genuine structure rather than memorised noise, an information-theoretic restatement of Occam's razor that shows up throughout model selection theory.

When it falls down

• The equivalence is about bit count, not compute. A billion-parameter LLM "compressing" a byte stream needs a full forward pass per byte or token, many orders of magnitude slower than a classical codec. The theorem says nothing about practical usability as a general-purpose zip replacement; it says achievable ratio equals predictive quality, full stop.
• The comparison is not apples to apples on cost. PNG and FLAC are tiny, fixed algorithms trained on nothing. The LLM in the experiment required internet-scale pretraining before it ever saw an image byte. "Beats PNG" is a true statement about compression ratio only; holding total engineering and compute cost fixed, the classical codec wins by an enormous margin.
• Practical LLM-as-compressor deployment needs bit-for-bit determinism. The decoder must reproduce the encoder's model outputs exactly, token by token, to invert the arithmetic code. Any nondeterminism, different hardware, different batching, non-associative floating point, breaks decoding. This is a real engineering obstacle, and it is the main reason nobody ships neural-network-based general compression in production despite the theoretical result.

Further reading

• Deletang et al., 2023, Language Modeling Is Compression, arXiv:2309.10668 - the LLM-as-arithmetic-coder experiment described above.
• MacKay, Information Theory, Inference, and Learning Algorithms - the free textbook treatment of arithmetic coding and the source coding theorem.
• Hutter Prize - the standing text-compression prize built on the compression-as-intelligence premise.