Everything Is Lossy Compression: A Rate-Distortion View of Quantisation, KV Caches, and Distillation

Four teams at the same company are trying to fit a model into less memory. One is quantising weights to 4 bits. One is evicting 80 percent of the KV cache. One is compressing prompts by dropping low-information tokens. One is distilling the 70B into an 8B. They read different papers, attend different workshops, and report different metrics: perplexity delta, attention recall, ROUGE against the uncompressed prompt, and win rate against the teacher.

They are solving the same problem. Each is choosing how many bits to keep and accepting a measurable degradation in exchange, and the theory of that exact trade has existed since Shannon published it in 1959, eleven years after the source coding theorem and largely unnoticed by the field that needs it most (Shannon, 1959, Coding Theorems for a Discrete Source with a Fidelity Criterion, IRE Int. Conv. Rec. 7:142-163).

Adopting the frame is not a relabelling exercise. It changes which comparisons are valid, exposes a systematic error in how compression results are reported, and identifies the one design decision that matters more than the algorithm: the choice of distortion measure.

Why this matters: Compression papers are compared by a single accuracy number at a single compression ratio. That comparison is meaningless unless both methods are evaluated under the same distortion measure at matched rate, and most of the time they are not. Rate-distortion gives you a principled way to say method A dominates method B, to say the two cross and where, and to notice when a reported win is an artefact of measuring the wrong thing.

TL;DR

• Every lossy compression choice is a point \((R, D)\): bits kept, and degradation incurred. Comparing two methods is only meaningful at matched rate under a shared distortion measure, which most published comparisons do not do.
• The rate-distortion function \(R(D)\) is convex and non-increasing, so the operating points that matter sit at the knee, and mixing two codecs can never beat the lower envelope of their curves.
• For a Gaussian source under squared error, \(D = \sigma^2 2^{-2R}\): each extra bit quarters the squared error. This is why the gap between 4-bit and 3-bit is far larger than the gap between 16-bit and 8-bit.
• The distortion measure is the design decision, not the algorithm. Weight MSE, layer-output error, KL to the original distribution, and end-task loss rank the same methods differently, and only the last one is what anyone actually wants.
• Dettmers and Zettlemoyer ran over 35,000 experiments across 19M to 176B parameters and found 4-bit is close to universally optimal for zero-shot accuracy at fixed total model bits (arXiv:2212.09720).
• Kumar et al. complicate that: post-training quantisation degradation grows with the amount of pretraining data, so more tokens can eventually make a model harder to compress and net-worse after quantisation (arXiv:2411.04330, ICLR 2025). The field has not resolved the tension.
• Average distortion is a weak guarantee. \(\mathbb{E}[d] \le D\) permits rare catastrophic outputs, and rare catastrophic outputs are what users notice. Reporting mean perplexity delta hides exactly the failure that matters.
• The rate axis is measured in the wrong unit almost everywhere. Compression ratios are not rates; bits per parameter and bits per token of context are, and they are the only figures that compare across methods.

At a Glance

flowchart LR
    S["Source<br/>weights, KV cache, prompt, teacher model"] --> E["Encoder<br/>quantiser, evictor, summariser, student"]
    E --> R["Rate<br/>bits retained"]
    E --> Rec["Reconstruction"]
    Rec --> D["Distortion<br/>measured against a chosen d(x, x-hat)"]
    R --> P["Operating point (R, D)"]
    D --> P
    P --> C{"Does method A dominate B?"}
    C -->|"Below everywhere"| W["A wins outright"]
    C -->|"Curves cross"| X["A wins in one regime only<br/>the crossing is the decision"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class S blue
    class E,Rec purple
    class R,D teal
    class P,C amber
    class W,X emerald

Before Anyone Called It Compression

Shannon's 1948 source coding theorem answered the lossless question: the minimum rate for exact reconstruction is the entropy. That theorem does not apply to anything in this article, because nothing here reconstructs anything exactly, and by 1959 Shannon had noticed the omission himself.

His fidelity-criterion paper introduced a distortion measure \(d(x, \hat{x})\) and defined the smallest rate achievable subject to an average distortion budget. Berger's 1971 monograph turned it into a field, and in 1972 Blahut and Arimoto independently found the alternating algorithm that computes \(R(D)\) for a known discrete source, the mirror image of the same algorithm for channel capacity.

timeline
    title Lossy Compression, From Shannon to Language Models
    1948 : Shannon's source coding theorem fixes the lossless rate at the entropy
         : Says nothing about approximate reconstruction
    1959 : Shannon adds a fidelity criterion and defines R of D
         : Lossy compression becomes a solved optimisation, in principle
    1972 : Blahut and Arimoto give an algorithm to compute R of D
         : Mirror image of the channel capacity algorithm
    2015 : Hinton, Vinyals and Dean formalise knowledge distillation
         : Soft targets as a distortion measure on distributions, not labels
    2017 : Balle et al. train image codecs end to end on a rate-distortion loss
         : The Lagrangian D plus lambda R becomes a training objective
    2018 : Alemi et al. decompose the ELBO into rate and distortion
         : Identical ELBO, different points on the curve, different behaviour
    2022 : LLM.int8, GPTQ and SmoothQuant make post-training quantisation practical
         : Distortion measured on layer outputs, not on weights
    2023 : H2O, StreamingLLM and LLMLingua compress caches and prompts
         : The same trade applied to activations and to text
    2024 : Precision-aware scaling laws tie quantisability to pretraining tokens
         : Rate and training compute stop being independent axes

The lineage worth noticing runs through neural image compression. Ballé, Laparra, and Simoncelli trained an autoencoder directly on the Lagrangian \(D + \lambda R\), treating rate as a differentiable quantity estimated by an entropy model, and beat JPEG2000 on both PSNR and perceptual measures (Ballé et al., 2017, End-to-end Optimized Image Compression, arXiv:1611.01704, ICLR 2017). That community has treated rate-distortion as an objective for a decade. The LLM compression community has treated it, at most, as an analogy.

How the Trade Actually Works

The function, and what it constrains

Fix a source \(X\), a reconstruction \(\hat{X}\), and a distortion \(d(x, \hat{x}) \ge 0\). The rate-distortion function is

a minimisation of mutual information over all channels meeting the distortion budget. Shannon's theorem says \(R(D)\) is exactly the asymptotically achievable rate: no code beats it, and codes approaching it exist.

Three structural properties do all the practical work.

Non-increasing. More tolerance never costs more bits. Obvious, and it means a method that spends more bits and incurs more distortion than another is strictly dominated, which happens more often in published comparisons than you would expect.

Convex. The curve bends toward the origin, so the interesting operating points sit at the knee, and time-sharing between two codecs traces a straight line between their points that can never beat the curve. If your measured points are non-convex, you have a suboptimal family, not an interesting discovery.

\(R(0) = H(X)\) for a discrete source under a distortion that vanishes only on exact matches. Lossless coding is the left endpoint of the same curve, not a separate subject. This is what makes "language modelling is compression" and "quantisation" the same conversation rather than adjacent ones (Delétang et al., 2024, arXiv:2309.10668).

For a Gaussian source under squared error the function is closed-form:

Each additional bit quarters the squared error. Read in reverse, each bit removed quadruples it. That geometric structure, not any property of neural networks, is the reason the 4-bit-to-3-bit step is where quantisation methods start visibly failing while the 16-bit-to-8-bit step is nearly free.

[IMAGE: A convex R(D) curve with three measured operating points plotted as dots: one on the curve, one above it labelled "achievable but suboptimal", and one below it labelled "impossible, check your measurement". Draw the straight time-sharing chord between two points to show it never dips below the curve. Caption: "Convexity is a sanity check as well as a theorem: non-convex measured points mean a suboptimal method family, not a discovery."]

Choosing the distortion measure is the whole design

This is where rate-distortion analysis of neural systems usually goes wrong, and it goes wrong in a way that produces confidently reported nonsense.

Weight MSE is the naive choice and nearly meaningless. Two quantisers with identical \(\lVert W - \hat{W}\rVert^2\) can differ by several points of downstream accuracy, because the error that matters is error in the function, and weight space and function space are related by the data distribution.

Layer output error, \(\mathbb{E}_x \lVert Wx - \hat{W}x\rVert^2\) over a calibration set, is what the good methods actually minimise. It weights each weight-space direction by how much the data excites it, so directions the input distribution never touches are free to be wrong. GPTQ is exactly this: a layer-wise second-order solve that quantises weights one column at a time while updating the remainder to compensate, which lets it quantise a 175B model to 3 or 4 bits in about four GPU hours with negligible accuracy loss (Frantar et al., 2023, GPTQ, arXiv:2210.17323, ICLR 2023). AWQ makes the same observation about excitation and acts on it differently, protecting the salient one percent of weight channels identified from activation magnitudes (Lin et al., 2024, arXiv:2306.00978, MLSys 2024).

KL to the original output distribution is the right measure when the deliverable is a distribution rather than an argmax, and it is what distillation minimises. Hinton, Vinyals, and Dean's soft-target formulation is a distortion measure on distributions, and the temperature parameter is a knob on how much of the teacher's tail structure counts as signal (Hinton et al., 2015, arXiv:1503.02531).

End-task loss is the honest measure and the most expensive to evaluate, which is why nobody optimises against it directly and everybody reports it.

The variational autoencoder makes the stakes explicit. Alemi et al. showed the ELBO decomposes into a rate term and a distortion term, so models with identical ELBO can sit at completely different points on the rate-distortion plane with qualitatively different behaviour (Alemi et al., 2018, Fixing a Broken ELBO, arXiv:1711.00464, ICML 2018). Optimising the sum tells you nothing about where on the curve you landed, and the same is true of any compression method reporting one scalar.

[IMAGE: Four scatter plots of the same 12 quantisation methods, ranked under four distortion measures: weight MSE, layer output error, KL to the original output distribution, and end-task accuracy. Connect each method's dot across panels to show the rank ordering crossing repeatedly. Caption: "Twelve methods, four measures, four different leaderboards."]

Why vector quantisation beats scalar quantisation, from theory

Shannon's theorem is asymptotic in block length. It promises codes approaching \(R(D)\) for long enough sequences and says nothing about quantising one scalar at a time, where the achievable rate can sit well above the bound. The gap has a name in classical coding theory, the space-filling loss, and for scalar quantisation of a Gaussian source it is about 0.25 bits per sample, or 1.53 dB.

Modern LLM quantisers claim it back by coding several weights jointly. QuIP# applies a randomised Hadamard transform to make the weights incoherent, which makes their joint distribution approximately spherical, and then quantises with codebooks built on the \(E_8\) lattice, which achieves the optimal sphere packing in eight dimensions (Tseng et al., 2024, QuIP#, arXiv:2402.04396, ICML 2024). That is not an ML heuristic; it is 1970s vector-quantisation theory applied to a new source, and it is why sub-4-bit weight quantisation became viable at all.

Seeing It in Motion

graph TD
    subgraph Sources["Four sources, one problem"]
        W["Model weights<br/>rate: bits per parameter"]
        KV["KV cache<br/>rate: bits per token of context"]
        PR["Prompt tokens<br/>rate: tokens retained"]
        TE["Teacher model<br/>rate: student parameters"]
    end
    W --> M["Choose a distortion measure"]
    KV --> M
    PR --> M
    TE --> M
    M --> M1["Layer output error"]
    M --> M2["Attention recall"]
    M --> M3["KL to original output"]
    M --> M4["End task loss"]
    M1 --> OP["Plot (rate, distortion)"]
    M2 --> OP
    M3 --> OP
    M4 --> OP
    OP --> V["Only now is a comparison meaningful"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class W,KV,PR,TE blue
    class M,M1,M2,M3,M4 purple
    class OP teal
    class V emerald

[IMAGE: A single rate-distortion plane with four labelled clusters of operating points, one per compression family: weight quantisation (2-16 bits per parameter), KV cache compression, prompt compression, and distillation. Axes: bits retained (log) and end-task loss delta. Draw the convex lower envelope through all of them. Caption: "Four literatures, one plane. The envelope is what a serving system actually gets to choose from."]

The value of the shared plane is that it makes a specific question answerable: given a fixed memory budget, which combination of compressions minimises degradation? That is a constrained optimisation over the joint curve, and it is not the question any individual paper answers, because each paper varies one axis with the others fixed at full precision.

stateDiagram-v2
    [*] --> FullPrecision
    FullPrecision: BF16 weights, BF16 cache, full prompt
    FullPrecision --> WeightQuant: cheapest bits first
    WeightQuant: INT4 weights, 4x on the constant term
    WeightQuant --> CacheQuant: context grows
    CacheQuant: INT4 or INT8 cache, 4x on the term that scales with context
    CacheQuant --> Eviction: context grows further
    Eviction: drop low-attention entries, lossy in a different way
    Eviction --> PromptCompress: input itself is redundant
    PromptCompress: remove low-information tokens before the model sees them
    PromptCompress --> [*]
    Eviction --> Catastrophic: past the knee
    Catastrophic: rare severe failures, invisible in mean metrics

The ordering in that diagram is not arbitrary. Weight bits are a constant cost paid once; cache bits scale with batch times context. At short context the weights dominate the budget and quantising them is the highest-leverage move; past a few thousand tokens of context the cache dominates and weight quantisation stops mattering. The correct compression strategy is a function of your context length, which is a claim the individual literatures cannot make because each holds the other axis fixed.

By the Numbers

Sources, in order: Frantar et al., 2023; Lin et al., 2024; Tseng et al., 2024; Dettmers et al., 2022; Xiao et al., 2023; Zhang et al., 2023; Xiao et al., 2024; Jiang et al., 2023; Delétang et al., 2024. Every figure is the authors' own reported result on their own evaluation, which is exactly the problem this article is about: the rate axes are different units, the distortion measures are incomparable, and no two rows can be placed on one curve without re-running both.

[IMAGE: Two stacked memory bars for Llama-3-8B on an 80 GB card at 8k context, batch 56. Top: BF16 everything, 15 GiB weights and 56 GiB cache. Bottom: INT4 weights and INT4 cache, 3.75 GiB and 14 GiB, with the freed space shaded and labelled "+224 more sequences". Caption: "The constant term and the term that scales, drawn to scale."]

Where credible people disagree. Dettmers and Zettlemoyer's conclusion, from more than 35,000 experiments across 19M to 176B parameters and four model families, is that 4-bit is close to universally optimal for zero-shot accuracy at fixed total model bits (Dettmers & Zettlemoyer, 2023, arXiv:2212.09720). Kumar et al. find that post-training quantisation degradation increases with the number of pretraining tokens the model saw, to the point that additional pretraining data can be net harmful if the model will be quantised afterwards (Kumar et al., 2025, Scaling Laws for Precision, arXiv:2411.04330, ICLR 2025).

Both results are careful and they are in tension. The reconciliation, in rate-distortion language, is that the source has changed: a model trained on far more tokens has different weight statistics, packing more information into the same parameter count and leaving less redundancy to discard. The optimal rate is a property of the source, and heavier training makes the source less compressible. That reading is consistent with both papers and is not, as of early 2026, settled.

A Concrete Example

Llama-3-8B served at 8,192 tokens of context. Compute the rate in each of the two places bits are spent, then compare three interventions at matched savings.

Step 1: establish the two rate axes. Weights are a constant: 8.03 billion parameters at 16 bits is 16.1 GB. The KV cache scales with batch and context: with grouped-query attention at 8 KV heads, head dimension 128, and 32 layers, each token of context costs

so a single 8,192-token sequence carries exactly 1 GiB of cache.

Step 2: find where the budget actually goes. On an 80 GB card, weights take 15.0 GiB and about 59 GiB is available for cache, which at 1 GiB per sequence is roughly 56 concurrent sequences. Total memory: 15.0 GiB of weights against 56 GiB of cache. The cache is 79 percent of the budget. Any effort spent on weight compression is optimising the smaller fifth.

Step 3: intervention A, quantise the weights to 4 bits. Rate falls from 16 to 4 bits per parameter, freeing 11.25 GiB. Cache capacity rises from 56 to 67 sequences, a 20 percent throughput gain. Distortion, measured as end-task accuracy, is small at 4 bits on this model class by the Dettmers and Zettlemoyer result.

Step 4: intervention B, quantise the cache to 4 bits. Rate falls from 16 to 4 bits per cached element, so per-token cost drops from 128 KiB to 32 KiB. At the same 59 GiB, cache capacity rises from 56 to 224 sequences, a 4x throughput gain, four times better than intervention A for the same nominal bit-width reduction. The reason is arithmetic, not cleverness: the same multiplicative saving applied to a term four times larger.

Step 5: intervention C, evict 75 percent of cache entries. Keep 25 percent of tokens by an attention-score heuristic. Per-sequence cache falls from 1 GiB to 256 MiB, so capacity also rises roughly 4x. Interventions B and C sit at the same rate and are therefore directly comparable, which is the point of the exercise.

Step 6: notice they are not the same distortion. Quantising the cache degrades every token slightly and uniformly. Eviction leaves 25 percent of tokens perfect and removes 75 percent entirely. Under mean perplexity these can look identical. Under a needle-in-a-haystack retrieval evaluation they are nothing alike: quantisation retains a degraded copy of the needle, while eviction may have deleted it. Same rate, same mean distortion, opposite tail behaviour.

Step 7: combine, and check the envelope. 4-bit weights plus 4-bit cache: 3.75 GiB of weights and 32 KiB per token, giving about 70 GiB for cache and roughly 280 sequences, a 5x gain over the baseline. The distortions compound, and nothing in either paper tells you how, because each measured its method against a full-precision everything-else.

Step 6 is the finding worth carrying. Two methods at identical rate and identical average distortion can have completely different tail behaviour, and average distortion is the only thing Shannon's formulation constrains. That is not a flaw in the theory; it is the theory telling you that your distortion measure is underspecified.

[IMAGE: Two histograms of per-example task-loss increase at the same compression rate, one for cache quantisation (narrow, centred just above zero) and one for cache eviction (a large spike at zero plus a long right tail). Mark the identical means with a shared vertical line. Caption: "Same rate, same mean, different product. The mean is the only thing the theory constrains."]

Where It Breaks

The asymptotic promise does not apply to your tensor

\(R(D)\) is achievable in the limit of long blocks. Element-wise scalar quantisation of a 4096-by-4096 matrix operates at block length one, where the achievable rate sits above the bound by the space-filling loss. Vector quantisation closes part of that gap and lattice codebooks close more, but no practical method reaches the bound, and quoting \(R(D)\) as if it were attainable overstates what is available.

The source is neither known nor stationary

The theory assumes a fixed, known \(p(x)\). Weight distributions differ substantially across layers, attention projections differ from MLP projections, and activation distributions shift with every input. This is not a footnote; it is the reason per-channel and per-group scales beat a single global scale by far more than the extra scale parameters cost, and the reason calibration set choice measurably changes post-training quantisation results.

Activation outliers are the sharpest version. A handful of feature dimensions in large transformers carry values orders of magnitude larger than the rest, which destroys any uniform quantisation grid. LLM.int8() handles them by decomposing the matmul and keeping outlier dimensions in 16 bits (Dettmers et al., 2022, arXiv:2208.07339); SmoothQuant migrates the difficulty from activations into weights by a per-channel rescaling that leaves the product unchanged (Xiao et al., 2023, arXiv:2211.10438, ICML 2023). Both are admissions that the source is not the well-behaved thing the theory assumes.

[IMAGE: Histogram of activation magnitudes for one transformer layer on a log-x axis, showing the bulk of dimensions clustered tightly and a handful of outlier feature dimensions two to three orders of magnitude to the right. Overlay a uniform INT8 quantisation grid sized to the outliers, showing the bulk collapsing onto one or two levels. Caption: "A uniform grid sized for the outliers wastes every level on the bulk. This is what SmoothQuant migrates and LLM.int8() routes around."]

Average distortion is the wrong guarantee for a product

\(\mathbb{E}[d] \le D\) permits rare catastrophic errors. A quantised model that is fine on 99.9 percent of inputs and produces garbage on the remaining tenth of a percent has excellent average distortion and is unshippable. Every serious compression evaluation eventually adds a tail quantile, at which point it has left Shannon's formulation for something empirical, and it should say so rather than continuing to invoke the theory.

Compression ratios are not rates, and the confusion is everywhere

"10x compression" is not a rate. Compared against what baseline precision, on which axis, holding what else fixed? A prompt compressor reporting 20x has changed the number of tokens; a weight quantiser reporting 4x has changed bits per parameter; a KV compressor reporting 5x may have changed either bits per element or number of elements. These are not commensurable, and the widespread habit of putting them in the same table, as the By the Numbers section above does deliberately, produces comparisons that mean nothing.

The fix is unglamorous and works: report bits per parameter and bits per token of context as absolute numbers, and report distortion under at least one measure another paper also used.

Rate interacts with training compute

The Kumar et al. result breaks a convenient assumption: that compression is a post-hoc transformation of a fixed source. If training longer makes a model less quantisable, then rate and pretraining compute are coupled, and the compute-optimal training recipe depends on the deployment precision. Nobody currently plans training runs this way, and the result suggests they should.

Composition is unstudied

Every paper varies one axis. Real deployments quantise weights and the cache and compress prompts and serve a distilled model. Whether these distortions compose additively, multiplicatively, or catastrophically is an empirical question with almost no published answer, and the worked example above ran into it immediately.

Alternative Designs

The last row is the one worth ending on. GQA and MLA are not post-hoc compression; they change the source so that less information is generated in the first place. In rate-distortion terms, they move the curve rather than moving along it, which is strictly better and available only to whoever trains the model. Every other row in the table is what everyone else has to work with.

How It Is Used in Practice

Choose the axis by context length, not by habit. The worked example gives the rule: below roughly a thousand tokens of context the weights dominate the memory budget and weight quantisation is the leverage; above a few thousand the cache dominates and cache compression is worth four to five times more. Teams routinely spend a quarter on weight quantisation for a long-context product where it changes very little.

[IMAGE: Decision chart with context length on the x-axis from 128 to 128k tokens on a log scale, and two shaded regions showing which term dominates the memory budget: weights below roughly 1k tokens, KV cache above. Annotate the crossover and label each region with its indicated intervention. Caption: "Which compression is worth doing is a function of context length, and no single paper's experiment tells you where you sit."]

Report both axes as absolute numbers. Bits per parameter and bits per token of context, not ratios. This single reporting change makes two papers comparable and costs nothing.

Evaluate distortion where the product lives. A summarisation service should measure distortion on summarisation quality; a code assistant on pass rate; a retrieval system on needle recall. Perplexity delta is a proxy, and the worked example shows it is a proxy that cannot distinguish two methods with opposite failure behaviour.

Measure the tail explicitly. Report the 95^th and 99^th percentile degradation alongside the mean. Compression failures are heavy-tailed by construction, and the mean is engineered to hide them.

Treat vendor compression claims as claims. "4x smaller with no quality loss" is a statement about a rate and a distortion measure, and it is incomplete until both are named. Ask which benchmark, at what percentile, against what baseline precision.

Insights Worth Remembering

1. The distortion measure is the design decision. The algorithm is downstream of it. GPTQ, AWQ and QuIP# differ mainly in what they consider an error, and their relative ranking changes with the measure you evaluate them under, which is why the literature looks more contentious than it is.

2. Compression ratios are not rates. A rate is bits per unit of source. Until both methods are expressed in the same unit against the same baseline, comparing them is not a comparison. This one habit invalidates a large fraction of published compression tables.

3. Each bit quarters the squared error, so the last bits are the expensive ones. \(D = \sigma^2 2^{-2R}\) is not a neural network fact, it is a Gaussian fact, and it predicts the observed cliff between 4-bit and 3-bit better than any architecture-specific story.

4. Two methods at the same rate and the same mean distortion can be entirely different products. Quantisation degrades everything slightly; eviction destroys some things completely. Average distortion is the only quantity Shannon's formulation constrains, and it is not the quantity users experience.

5. Compress the term that scales, not the term that is constant. Weights are a fixed cost; the KV cache scales with batch times context. Long-context deployments that quantise weights and leave the cache in BF16 have optimised a fifth of their memory.

6. Vector quantisation is not a trick, it is the theory. The bound is asymptotic in block length, so scalar quantisation is provably suboptimal by a known margin. QuIP#'s \(E_8\) lattice is 1970s coding theory recovering exactly that margin.

7. A more heavily trained model may be a less compressible source. If Kumar et al.'s result holds up, quantisability is something you build in during pretraining rather than something you discover afterwards, and the compute-optimal recipe depends on the deployment precision.

8. Lossless and lossy are endpoints of one curve. \(R(0) = H(X)\), so "a language model is a compressor" and "a quantised language model is a lossy compressor" are statements about the same object at two operating points, and the interesting question in both cases is whether you are charging for the model.

Open Questions

How do stacked compressions compose? Everyone quantises weights and the cache and compresses prompts simultaneously; nobody has published a systematic study of the joint distortion surface. Whether the errors are roughly independent or interact badly is measurable with existing tools and, as of early 2026, largely unmeasured.

Is there a usable operational \(R(D)\) for a transformer? Computing the true function requires the source distribution and is out of reach. Whether a useful estimate exists, perhaps per-layer and under an activation-weighted distortion, would turn a qualitative frame into a design tool that says how many bits a given layer can afford. Nothing published does this convincingly.

What is the right distortion measure for agentic use? All the standard measures score a single output. Agents chain dozens of model calls, so a small per-step distortion compounds over a trajectory in a way no per-token measure captures. Compression evaluated on single-turn benchmarks may be badly mis-estimating its cost in agent deployments, and this is speculation rather than a measured claim.

Does the Dettmers and Kumar tension resolve in favour of one recipe? The "4-bit is optimal" and "more pretraining data hurts quantisability" results are both careful and point in different directions for a lab planning a training run. Resolving it requires precision-aware scaling experiments at frontier scale, which only a handful of organisations can run.

Can rate-distortion become the training objective, as it did for images? Neural image codecs train on \(D + \lambda R\) end to end and dominate hand-designed codecs. Quantisation-aware training is the closest LLM analogue and it optimises distortion at a fixed rate rather than the Lagrangian. Whether a learned, differentiable rate model over weights would beat the current fixed-grid approaches is, as far as published work goes, untested at scale.

Sources and Further Reading

Foundational theory

1. Shannon, C. E. (1959). "Coding Theorems for a Discrete Source with a Fidelity Criterion." IRE International Convention Record, 7, 142-163. PDF
2. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory, 2^nd ed. Wiley. Chapter 10 covers rate-distortion; Chapter 3 covers typical sets.
3. Berger, T. (1971). Rate Distortion Theory: A Mathematical Basis for Data Compression. Prentice-Hall.

Rate-distortion in machine learning

4. Ballé, J., Laparra, V., & Simoncelli, E. P. (2017). "End-to-end Optimized Image Compression." ICLR. arXiv:1611.01704
5. Alemi, A. A., Poole, B., Fischer, I., Dillon, J. V., Saurous, R. A., & Murphy, K. (2018). "Fixing a Broken ELBO." ICML. arXiv:1711.00464
6. Delétang, G., Ruoss, A., Duquenne, P.-A., et al. (2024). "Language Modeling Is Compression." ICLR. arXiv:2309.10668
7. Hinton, G., Vinyals, O., & Dean, J. (2015). "Distilling the Knowledge in a Neural Network." arXiv:1503.02531

Weight and activation quantisation

8. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." NeurIPS. arXiv:2208.07339
9. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR. arXiv:2210.17323
10. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2023). "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models." ICML. arXiv:2211.10438
11. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., & Han, S. (2024). "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys. arXiv:2306.00978
12. Tseng, A., Chee, J., Sun, Q., Kuleshov, V., & De Sa, C. (2024). "QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks." ICML. arXiv:2402.04396
13. Dettmers, T., & Zettlemoyer, L. (2023). "The case for 4-bit precision: k-bit Inference Scaling Laws." ICML. arXiv:2212.09720
14. Kumar, T., Ankner, Z., Spector, B. F., et al. (2025). "Scaling Laws for Precision." ICLR. arXiv:2411.04330

Cache and prompt compression

15. Zhang, Z., Sheng, Y., Zhou, T., et al. (2023). "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS. arXiv:2306.14048
16. Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2024). "Efficient Streaming Language Models with Attention Sinks." ICLR. arXiv:2309.17453
17. Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023). "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models." EMNLP. arXiv:2310.05736

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.