Language Modelling Is Compression: The Seventy-Year-Old Idea Underneath Every LLM

In 1951, Claude Shannon published the results of a parlour game. He would show a human subject a fragment of English text, cut off mid-sentence, and ask them to guess the next letter; wrong guesses were counted and the game repeated, letter after letter, through passages of prose. From the statistics of those guesses he derived bounds on the entropy of printed English: somewhere between 0.6 and 1.3 bits per letter, far below the 4.7 bits that a 27-symbol alphabet would need if letters were random (Shannon, 1951, Prediction and Entropy of Printed English, Bell System Technical Journal 30(1), 50–64). The experiment needed no computer. It needed a predictor, and the best predictor of English available in 1951 was a person.

Seventy-two years later, a team at DeepMind ran the same experiment with a different predictor. Chinchilla 70B, wired to an arithmetic coder, losslessly compressed a gigabyte of Wikipedia to 8.3% of its original size, where gzip manages 32.3% and the specialist tools built by the data-compression community over decades do somewhat better than 11% (Delétang et al., 2023, Language Modeling Is Compression, ICLR 2024, arXiv:2309.10668). The same model, trained almost entirely on text, also compressed ImageNet patches to 43.4% against PNG's 58.5%, and LibriSpeech audio to 16.4% against FLAC's 30.3%. Nobody retrained anything. Prediction was the only capability involved, because prediction is the only capability compression ever needed.

Why this matters: The equivalence between language modelling and lossless compression is exact, bidirectional, and old. It explains why cross-entropy loss is the number every lab optimises, gives a tokenizer-independent way to compare models, underwrites a benchmark that is hard to contaminate, and is the cleanest formal argument that "predicting the next token" is not a shallow trick. Anyone who works with LLMs is working with a compressor, whether they think of it that way or not.

TL;DR

• A language model plus an arithmetic coder is a lossless compressor whose output length is the model's log-loss on that text, within about two bits total; minimising cross-entropy and maximising compression are the same act, not analogous acts (Delétang et al., 2023).
• The equivalence runs both ways: any lossless compressor induces a probability distribution over sequences, so gzip is a (bad) language model, and a kNN classifier built on gzip distances briefly appeared to beat BERT before an accuracy-reporting bug was found (Jiang et al., 2023; Schutte, 2023).
• Chinchilla 70B compresses enwik9 to 8.3% (0.664 bits per byte) versus gzip's 32.3% and LZMA2's 23.0%, and beats PNG and FLAC on images and audio it was never trained on, in-context learning doing the domain adaptation (Delétang et al., 2023).
• The catch is model size: counted honestly, 140 GB of weights dwarf the bytes saved on a 1 GB file, so a pretrained LLM is only a real compressor at corpus scales where the weights amortise; the Hutter Prize's CPU-and-size rules are designed around exactly this accounting.
• Compression performance predicts intelligence unusually well: across 31 public LLMs, average bits-per-character on fresh corpora correlates almost linearly with benchmark scores in the matching domain (Huang et al., 2024, Compression Represents Intelligence Linearly, COLM 2024).
• Bits per byte, the compression-native metric, is the tokenizer-invariant way to compare language models, which perplexity is not (see bits per byte).

At a Glance

The whole construction fits in one loop: the model supplies a probability distribution for the next symbol, the arithmetic coder turns that distribution plus the actual symbol into a sliver of the output bitstream, and the identical model on the receiving end inverts the process. The model is never modified; the coder is model-agnostic; the interface between them is nothing but \(p(x_t \mid x_{<t})\).

flowchart LR
  Text["Input bytes"] --> Model["Language model"]
  Model --> Dist["Next-symbol distribution"]
  Dist --> Coder["Arithmetic coder"]
  Coder --> Bits["Compressed bitstream"]
  Bits --> Decoder["Arithmetic decoder"]
  Model2["Identical model"] --> Decoder
  Decoder --> Out["Original bytes, exactly"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  class Text blue
  class Model,Model2,Dist purple
  class Coder,Decoder purple
  class Bits,Out teal

[IMAGE: A two-panel diagram. Left panel: Shannon's 1951 guessing game, a hand-drawn table of a sentence with the number of guesses needed per letter written above each character. Right panel: the same sentence with a modern LLM's per-token log-probabilities written above each token. Caption: "Same experiment, different predictor: guesses per letter and bits per token are both measurements of surprise."]

From Guessing Games to Gigabytes

The theory came first, and it came essentially complete. Shannon's 1948 paper defined entropy as the irreducible average number of bits needed to communicate a source's output, and proved that a code matched to the true distribution attains it while any mismatched code pays a penalty of exactly the KL divergence between truth and belief (Shannon, 1948, A Mathematical Theory of Communication). The 1951 guessing experiment then showed how to estimate that limit for natural language using whatever predictor you have. What 1951 lacked was a practical way to cash in a good predictor for actual compressed bits: Huffman coding, published the following year, wastes up to a bit per symbol because it must assign whole-bit codewords, which is catastrophic when the ideal cost of a well-predicted symbol is 0.05 bits.

Arithmetic coding removed that obstacle. Developed through the 1970s from Rissanen's work at IBM and made practical for the working programmer by Witten, Neal, and Cleary's 1987 implementation paper, it encodes an entire sequence as a single number in a shrinking interval, charging each symbol its exact information content, fractional bits included (Witten, Neal, and Cleary, 1987, Arithmetic Coding for Data Compression, Communications of the ACM 30(6)). From 1987 onward, lossless compression had a clean division of labour: the coder is solved, so all remaining progress is better prediction. The compression community internalised this decades before deep learning arrived; PAQ and its descendants, the perennial record-holders, are ensembles of context models feeding an arithmetic coder, gradient-updated online, a design any modern ML practitioner would recognise immediately.

The theoretical wing of the same idea went further. Solomonoff's induction (1964) and Kolmogorov's complexity (1965) defined the ideal predictor as the shortest program reproducing the data, making "find the best compression" the formal definition of inductive inference; Hutter's AIXI later built a theory of universal intelligent agents directly on top of it. That lineage is why, in 2006, Marcus Hutter put up prize money for compressing Wikipedia, on the explicit thesis that text compression is an AI-complete measure of machine intelligence; the contest, now with a fund of half a million euros and running on enwik9, has been won in small increments by context-mixing programs, most recently fx2-cmix in September 2024 (Hutter Prize).

The neural era closed the loop. Fabrice Bellard's NNCP demonstrated a transformer trained from scratch, online, during compression reaching about 110 MB on enwik9, state of the art among practical compressors, with no pretrained weights to account for (Bellard, 2021, NNCP v2: Lossless Data Compression with Transformer). Delétang et al. then showed what happens when you drop a pretrained frontier model into the same harness, and Huang et al. inverted the question: instead of using models to compress, use compression to measure models.

timeline
    title Prediction and compression, converging for seventy years
    1948 : Shannon defines entropy and the cost of wrong beliefs
    1951 : The guessing game bounds English at 0.6 to 1.3 bits per letter
    1987 : Witten, Neal and Cleary make arithmetic coding practical; prediction becomes the whole game
    2006 : Hutter Prize stakes money on compression as intelligence
    2021 : Bellard's NNCP puts an online-trained transformer at the top of the enwik9 rankings
    2023 : Chinchilla 70B compresses text, images and audio past every specialist tool
    2024 : Compression shown to track benchmark intelligence almost linearly; fx2-cmix takes the prize record

The Equivalence, Made Mechanical

One direction: a predictor becomes a compressor

Arithmetic coding maintains an interval \([l, h) \subset [0, 1)\), initially the whole unit interval. To encode symbol \(x_t\), partition the current interval according to the model's distribution \(p(\cdot \mid x_{<t})\), and shrink to the sub-interval corresponding to the symbol that actually occurred. After the last symbol, emit enough bits to name a number inside the final interval. Each step multiplies the interval's width by \(p(x_t \mid x_{<t})\), so the final width is \(\prod_t p(x_t \mid x_{<t})\), and naming a number inside an interval of width \(w\) takes at most \(\lceil -\log_2 w \rceil + 1\) bits. The total code length is therefore

and the sum on the right is precisely the model's log-loss on the sequence: the training objective, evaluated on this text.

[IMAGE: Side-by-side mirrored flow diagram: encoder on the left (model, distribution, interval shrink, emitted bits), decoder on the right as an exact mirror, with a single thin bitstream connecting them and a large shared "identical model" block spanning both sides. Caption: "The decoder is the encoder run in reverse; the only shared secret is the model."] Two bits of overhead, total, for the entire file. This is why the claim is not a metaphor. The cross-entropy that every training run minimises is the expected compressed length per symbol under this scheme, and a loss curve is a compression-ratio curve with the axis relabelled. Every scaling-law plot you have seen is a statement about compression (see language modelling as compression for the concept-length version).

The decoder needs no side channel: it holds the same model, reconstructs the same partition of the current interval, observes which sub-interval the encoded number falls in, recovers the symbol, updates the context, and repeats. All that is required is that encoder and decoder compute bit-identical probabilities. Hold that thought for the failure-modes section.

The other direction: a compressor is a predictor

Any lossless compressor \(c\) assigns each string a code length \(|c(x)|\), and \(2^{-|c(x)|}\) is (after normalisation) a probability distribution: short codes mean high probability. So gzip defines a language model. It is a poor one, blind to anything its 32 KB window and literal-match machinery cannot see, but it is not nothing, and "not nothing" occasionally makes headlines. In 2023, a paper showed a k-nearest-neighbour classifier using gzip compression distances as its metric competing with BERT on text classification benchmarks (Jiang et al., 2023, "Low-Resource" Text Classification: A Parameter-Free Classification Method with Compressors, Findings of ACL 2023). The viral claim partially unravelled when Ken Schutte showed the evaluation was effectively reporting top-2 accuracy, counting a prediction correct if either of two tied neighbours was right; recomputed conventionally, the method fell back to earth on most datasets (Schutte, 2023, Bad numbers in the "gzip beats BERT" paper?). The episode is worth remembering in both directions: compression really does encode semantic similarity, and sloppy evaluation really can manufacture a revolution out of it.

What the numbers actually showed

The Delétang results reward a careful read. The headline is Chinchilla 70B reaching 0.664 bits per byte on enwik9, an 8.3% compression rate. The subtler findings are two. First, the cross-modal transfer: the same text-trained model beating PNG on ImageNet patches and FLAC on LibriSpeech is in-context learning doing real work, the model adapting to byte statistics it never saw in training, within a context of a few thousand bytes. Second, the accounting: the paper's "adjusted compression rate" adds the compressed size of the model itself to the numerator, and under that accounting every Chinchilla-class model is a catastrophically bad compressor, because 70B parameters dwarf any single-gigabyte file. A 3.2M-parameter transformer trained on enwik8 achieves a 17.7% adjusted rate on enwik9, beating gzip's 32.3% honestly; the 70B model cannot come close once its weights are on the bill. Scale helps raw prediction and murders the amortisation argument at benchmark sizes, which is exactly why the practical compression records belong to NNCP-style systems that train during compression and ship no weights at all.

Seeing It in Motion

Transmission is the cleanest way to see the construction whole: both parties hold the same model, and the only thing that crosses the wire is the arithmetic-coded residue of what the model could not predict.

sequenceDiagram
  participant S as Sender
  participant M1 as Model (sender copy)
  participant M2 as Model (receiver copy)
  participant R as Receiver
  S->>M1: Context so far
  M1-->>S: p(next symbol) as interval partition
  S->>R: Bits naming the true sub-interval
  R->>M2: Same context so far
  M2-->>R: Identical partition
  R->>R: Locate sub-interval, recover symbol
  Note over S,R: Repeat per symbol; well-predicted symbols cost fractions of a bit

The design space of compressors is then a spectrum of where the predictor comes from: fixed handcrafted statistics, online-learned statistics, or pretrained weights.

flowchart TB
  subgraph Classic["gzip (1992)"]
    A1["LZ77 match finder"] --> A2["Huffman coder"]
  end
  subgraph Mixing["cmix family"]
    B1["Hundreds of context models"] --> B2["Learned mixer, updated online"] --> B3["Arithmetic coder"]
  end
  subgraph Neural["NNCP / LLM-as-compressor"]
    C1["Transformer predictor"] --> C2["Arithmetic coder"]
  end
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  class A1,A2 slate
  class B1,B2 purple
  class B3,C2 teal
  class C1 purple

[IMAGE: Animated-style frame sequence of the unit interval narrowing over four encoded symbols: each frame shows the current interval subdivided into labelled regions sized by the model's probabilities, with the chosen region highlighted and zoomed into the next frame. Caption: "Arithmetic coding charges each symbol its exact information content: the interval shrinks by a factor of p at every step."]

By the Numbers

Sources: gzip, LZMA2, PNG, FLAC, and Chinchilla rows from Delétang et al.'s raw compression rates (arXiv:2309.10668); the paper's full table also has each specialist tool scored outside its home modality, and the collapse there (FLAC treating Wikipedia text as audio barely compresses it at all) is the mirror image of the LLM's transfer. NNCP figure from Bellard's report, including its dictionary (bellard.org/nncp). fx2-cmix from the Hutter Prize record set in September 2024 by Kaido Orav and Byron Knoll (prize.hutter1.net). The Chinchilla figure excludes model weights; the two records above it include everything needed to decompress, which is the honest comparison and the reason they, not the LLM, hold the records.

[IMAGE: Horizontal bar chart of the enwik9 column above, bars sorted from gzip down to Chinchilla 70B, with a hatched extension on the Chinchilla bar labelled "plus 140 GB of weights" extending far off the chart edge. Caption: "Raw rate flatters the pretrained model; adjusted rate reverses the ranking."]

A Concrete Example

Take a four-token message and a model with a 4,096-token vocabulary. The model assigns these conditional probabilities as the message unfolds:

1. Token "The", an unsurprising opener: \(p = 0.05\). Cost: \(-\log_2 0.05 = 4.32\) bits. The interval \([0, 1)\) shrinks to a sub-interval of width \(0.05\), say \([0.31, 0.36)\).
2. Token "cat", plausible continuation: \(p = 0.02\). Cost: \(5.64\) bits. Width is now \(0.05 \times 0.02 = 0.001\).
3. Token "sat", strongly predicted by the idiom: \(p = 0.40\). Cost: \(1.32\) bits. Width: \(4 \times 10^{-4}\).
4. Token "quietly", mildly surprising: \(p = 0.01\). Cost: \(6.64\) bits. Final width: \(4 \times 10^{-6}\).

Total ideal cost: \(4.32 + 5.64 + 1.32 + 6.64 = 17.92\) bits; the coder emits at most \(\lceil 17.92 \rceil + 1 = 19\) bits, so a two-and-a-half-byte encoding of a message whose UTF-8 form is 19 bytes: about 13% compression rate, Chinchilla-on-enwik9 territory. Now replay it with a uniform model, \(p = 1/4096\) per token: \(12\) bits each, \(48\) bits total, and the "compressor" expands typical text once framing overhead lands. Every byte of compression came from the gap between the model's beliefs and ignorance, which is the sense in which the compressed size is a measurement of the model.

Run the replay one more time with a better model, one that assigns "sat" \(0.8\) instead of \(0.4\) and "quietly" \(0.04\) instead of \(0.01\): total drops to \(14.6\) bits. That 3.3-bit saving is the exact, cashable value of the better model's knowledge on this message. No benchmark, judge, or rubric intervenes; the bits are the score.

Where It Breaks

The weights are part of the message

A compressor's honest size is everything the receiver needs: bitstream plus decompressor. For gzip that is kilobytes; for Chinchilla 70B it is on the order of 140 GB of weights. Amortised over the open web the weights vanish; over a 1 GB benchmark they are ruinous, and Delétang et al.'s adjusted rates make small from-scratch transformers the honest winners at that scale. The Hutter Prize's much-criticised CPU, memory, and time limits are not a quirk; they are a position in this exact argument, insisting the whole system fit the budget. Whether intelligence-relevant compression should count the weights, and over what corpus size the amortisation becomes fair, remains genuinely contested.

Bit-exact determinism, or garbage

Arithmetic decoding requires the receiver's probabilities to match the encoder's to the last bit; a difference of \(10^{-9}\) in one logit's softmax can shift a sub-interval boundary, decode one wrong symbol, and corrupt everything after it. Floating-point non-associativity across GPUs, kernel versions, and batch shapes makes this brittle in exactly the way LLM serving is nondeterministic in general (Thinking Machines Lab, 2025, Defeating Nondeterminism in LLM Inference). Practical neural compressors pin hardware and use integer or deterministic arithmetic in the coder; a research result measured on one machine does not automatically make a shippable file format.

[IMAGE: Log-log scatter plot of compression throughput (bytes per second, x-axis) against compression ratio on text (y-axis) with labelled points for zstd, gzip, LZMA2, cmix, NNCP, and a pretrained-70B system, showing a stark empty region where fast-and-tight would live. Caption: "The Pareto frontier of lossless text compression: six orders of magnitude of speed separate the ratio champions from the tools anyone actually runs."]

Throughput is off by six orders of magnitude

gzip moves hundreds of megabytes per second on one core. A 70B model doing a forward pass per token to compress a gigabyte is a GPU-days proposition, and even purpose-built practical systems trade this consciously: NNCP's record-setting runs took days on GPU hardware, and Bellard ships ts_zip as the "practical" variant with a smaller model at a few hundred kilobytes per second on a GPU. LLM-based compression is real; LLM-based fast compression is not, and the record tables segregate by resource class for that reason.

Contamination flatters, and the metric knows it

enwik9 is Wikipedia, and Wikipedia is in every pretraining mix, so part of Chinchilla's 8.3% is memory rather than modelling. The compression framing at least makes the problem crisp: evaluate on bytes created after the training cutoff and the flattery disappears, which is precisely how compression-based evaluations like Huang et al.'s stay resistant to the benchmark-contamination rot that plagues QA leaderboards (see reasoning evals and contamination). Uniqueness of the text matters as much as recency; fresh-but-templated text still compresses suspiciously well.

Chat tuning quietly damages the compressor

The equivalence prices a model's probabilities, and post-training reshapes those probabilities away from corpus statistics toward preferred behaviour. RLHF-tuned models are measurably worse calibrated as next-token predictors than their base checkpoints (Kadavath et al., 2022, arXiv:2207.05221), so the best available compressor is usually the base model, and a compression eval of an instruct model partly measures the alignment tax, not the knowledge. Anyone using bits-per-byte to rank deployed chat models is comparing objects the metric was not defined for.

Alternative Designs

The table's diagonal is the point: every step down buys ratio with compute and generality of the predictor, and the "best" compressor is entirely a statement about whose resources are on the bill. There is no regime where the frontier LLM is the right practical choice today; its column exists because it is the best measurement instrument, which is a different job.

How It Is Used in Practice

The working uses of the equivalence are mostly evaluative, and they are load-bearing. Bits per byte is the standard way to report language-modelling quality without the tokenizer gaming that makes perplexity incomparable across models: perplexity-per-token can be halved by merging tokens, while bits-per-byte holds still (see bits per byte). Compression-based evaluation extends this into a benchmark methodology: Huang et al. measured 31 open models' bits-per-character on Common Crawl, GitHub, and arXiv corpora and found benchmark ability in the matching domain (knowledge, code, mathematics) tracking compression almost linearly, tightly enough to use compression as a cheap, contamination-resistant proxy score (Huang et al., 2024, COLM). Related work uses compressibility of the training data itself as a difficulty measure, with gzip-estimated data complexity predicting how scaling-law coefficients shift across datasets (Pandey, 2024, gzip Predicts Data-dependent Scaling Laws, arXiv:2405.16684).

[IMAGE: Scatter plot of 31 language models: average bits-per-character on a fresh corpus (x-axis, reversed so better is rightward) against average benchmark score in the matching domain (y-axis), points coloured by model family, with a fitted line and a visibly tight spread. Caption: "Compression tracks capability almost linearly across model families, which is what makes it usable as a benchmark rather than a curiosity."]

The framing also does conceptual work inside the labs. Ilya Sutskever's 2023 argument that unsupervised learning succeeds because it is compression, and that better compression of the joint data forces the discovery of shared structure, is the modern restatement of Solomonoff's position with transformer-shaped evidence behind it. And the training objective itself is a minimum-description-length principle wearing a different name: a model that overfits memorises the bitstream instead of shortening it, and regularisation is the two-part-code tax made differentiable.

Actual deployment of neural lossless compression stays niche but real: NNCP-class systems hold records where rules allow GPUs; ts_zip compresses text files with a small LLM at ratios well beyond zstd for users who can spend GPU time; and floating-point-free variants of the arithmetic-coding harness are the standard trick in learned compression of images and genomics, fields that adopted the predictor-plus-coder split from the same theory.

Insights Worth Remembering

1. The equivalence is an identity, not an analogy. Code length equals log-loss plus two bits, by construction of the arithmetic coder. "LLMs are like compressors" understates it; the loss curve on your dashboard is denominated in compressed bits.
2. All compression progress since 1987 is prediction progress. The coder is optimal and model-agnostic, so gzip, cmix, NNCP, and Chinchilla differ only in the quality and cost of \(p(x_t \mid x_{<t})\). The data-compression community was doing sequence modelling before it had the vocabulary.
3. Shannon's parlour game was the first LLM eval. Human guessing bounded English at 0.6 to 1.3 bits per letter in 1951; frontier models now sit inside that human band on comparable text, and the measurement methodology has not fundamentally changed, only the predictor.
4. Raw and adjusted compression rates answer different questions. Raw rate measures the model's knowledge; adjusted rate measures a deployable artefact. The 70B model wins the first and loses the second catastrophically at gigabyte scale, and confusing the two is the most common way this topic is misused.
5. Compression is the benchmark that fights back against contamination. Score on newly created bytes and there is nothing to memorise in advance; the linear compression-to-capability relationship makes the score meaningful, not just clean.
6. In-context learning is measurable in bits. A text-trained model beating FLAC on audio within a few-thousand-byte context is adaptation you can price exactly: the per-byte cost falls as the context grows and the model updates its beliefs without touching a weight.
7. The equivalence is fragile in exactly one place: determinism. One non-reproducible logit breaks decoding entirely. The gap between "compresses in a paper" and "a file format" is the gap between a measurement and a contract.

Open Questions

• How far does the compression-intelligence line extend? The linear relationship is measured across base models on knowledge, code, and maths (Huang et al., 2024). Whether it holds through heavy post-training, tool use, and reasoning-RL, where the deployed system's ability diverges from its next-token probabilities, is open; the RLHF calibration damage suggests the line at least bends.
• What is the right amortisation scale for the weights? Adjusted-rate accounting is fair at 1 GB and arguably unfair at web scale, but nobody has a principled rule for where the crossover sits, and the answer determines whether "LLMs are the best compressors" is simply true, simply false, or scale-indexed.
• Can lossless ratios ever be bought at practical speed? The gap between zstd's throughput and NNCP's ratio spans six orders of magnitude in speed for a factor of roughly two in size. Speculative and batched decoding tricks narrow it; whether they narrow it enough for a mainstream format is an engineering question with no demonstrated answer.
• Does compression measure what lossy abstraction contributes? Human memory is aggressively lossy, and some argue intelligence lives in what is discarded. The lossless framing prices every byte, boilerplate and insight alike; whether a principled lossy variant (rate-distortion with a semantic distortion measure) could rank models better is speculative and unresolved.
• Is enwik-scale text still the right substrate? The Hutter Prize's thesis dates from an era when 1 GB of Wikipedia looked like "human knowledge". Models now compress it partly from memory, and the interesting signal may have moved to fresh multimodal streams, where the Delétang cross-modal results are the only substantial public data point.

Sources and Further Reading

1. Shannon, C. E. (1948). "A Mathematical Theory of Communication." Bell System Technical Journal, 27(3), 379–423. IEEE reprint
2. Shannon, C. E. (1951). "Prediction and Entropy of Printed English." Bell System Technical Journal, 30(1), 50–64. archive.org scan
3. Witten, I. H., Neal, R. M., & Cleary, J. G. (1987). "Arithmetic Coding for Data Compression." Communications of the ACM, 30(6), 520–540. ACM DL
4. Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., & Veness, J. (2023). "Language Modeling Is Compression." ICLR 2024. arXiv:2309.10668
5. Huang, Y., et al. (2024). "Compression Represents Intelligence Linearly." COLM 2024. arXiv:2404.09937
6. Bellard, F. (2021). "NNCP v2: Lossless Data Compression with Transformer." bellard.org/nncp
7. Hutter, M. (2006–present). "The Hutter Prize / Human Knowledge Compression Contest." prize.hutter1.net
8. Jiang, Z., Yang, M., Tsirlin, M., Tang, R., Dai, Y., & Lin, J. (2023). "'Low-Resource' Text Classification: A Parameter-Free Classification Method with Compressors." Findings of ACL 2023. ACL Anthology
9. Schutte, K. (2023). "Bad numbers in the 'gzip beats BERT' paper?" kenschutte.com
10. Pandey, R. (2024). "gzip Predicts Data-dependent Scaling Laws." arXiv:2405.16684
11. Kadavath, S., et al. (2022). "Language Models (Mostly) Know What They Know." arXiv:2207.05221
12. Thinking Machines Lab (2025). "Defeating Nondeterminism in LLM Inference." thinkingmachines.ai
13. Mahoney, M. (ongoing). "Large Text Compression Benchmark." mattmahoney.net/dc/text.html

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.