The Rate-Distortion View of the ELBO

Two models, same architecture, same dataset, same ELBO to three decimal places. One has a latent code that cleanly separates the factors of variation. The other has a latent carrying literally zero information about the input, with a decoder that has learned the marginal and ignores \(z\) entirely. The objective cannot tell them apart, because the ELBO is a sum of two terms and only their sum appears in the loss (Alemi et al., 2018, Fixing a Broken ELBO, arXiv:1711.00464).

The decomposition

Write the negative ELBO per data point as

\(D\) is the reconstruction cost in nats, the price of describing \(x\) once you know \(z\). \(R\) is the cost of describing \(z\) itself relative to the prior, which in a bits-back coding sense is exactly the channel capacity the encoder is using. Minimising \(-\mathcal{L}\) minimises \(R + D\), a single point on a line of slope \(-1\) in the \((R, D)\) plane, and every point on that line is equally good as far as the objective is concerned.

The two ends of the feasible region have names. The auto-decoding limit sits at \(R = 0\), where the posterior equals the prior, the latent is unused, and a sufficiently powerful decoder absorbs all the structure; distortion there equals the data entropy \(H\). The auto-encoding limit sits at \(D \to 0\) with \(R\) as large as the latent allows: near-perfect reconstruction paid for with a very informative code. The rate also brackets the mutual information the representation carries, with \(H - D \le I(X; Z) \le R\), which is what turns the picture from a metaphor into a bound.

Beta is a Lagrange multiplier

Once you see the frontier, the fix is obvious: pick your point on it explicitly. Weighting the KL term,

is constrained optimisation, minimising distortion subject to a rate budget, with \(\beta\) the multiplier. This is exactly the \(\beta\)-VAE objective, arrived at from representation learning rather than from coding (Higgins et al., 2017, beta-VAE, ICLR). Large \(\beta\) squeezes the rate and pushes toward auto-decoding; small \(\beta\) opens the channel. The disentanglement that made \(\beta\)-VAE famous is a consequence of forcing the encoder to spend a scarce bit budget on the factors that reduce distortion most.

That framing also explains why the representation-learning promise did not fully survive contact with evidence. Training over 12,000 models across seven datasets, Locatello and colleagues found no unsupervised method reliably produced disentangled representations, and proved that unsupervised disentanglement is impossible without inductive biases on both model and data (Locatello et al., 2019, Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, arXiv:1811.12359). The rate knob controls how many bits the code holds. It does not control which bits.

What decides where you land

Nothing in the ELBO picks a point on the frontier, so the architecture does. A decoder powerful enough to model \(p(x)\) on its own, an autoregressive one over pixels or tokens, can reach \(D = H\) with \(R = 0\), which is cheaper than paying any rate at all. That is posterior collapse, and it is the optimum rather than an optimisation failure (Chen et al., 2017, Variational Lossy Autoencoder, arXiv:1611.02731). Conversely a deliberately weakened decoder, a limited receptive field or a small conditional model, makes the latent the cheapest route to low distortion and pins the solution at high rate.

When it breaks

The frontier is model-dependent. The achievable \((R, D)\) region depends on the relative power of encoder, decoder and prior. Comparing \(\beta\) values across two architectures compares nothing.

Rate is not information. \(R\) upper-bounds \(I(X; Z)\) and the gap can be large. An encoder can burn rate encoding noise. Diagnose with per-dimension KL rather than the aggregate: a healthy code has many dimensions carrying a few tenths of a nat each, not two dimensions carrying everything.

Tuning \(\beta\) is not the only lever, and often not the best one. Free bits impose a floor on per-dimension KL and hold rate open where a global \(\beta\) would not. Discrete bottlenecks fix the rate by construction, since a codebook of size \(K\) per position caps it at \(\log K\) nats (van den Oord, Vinyals and Kavukcuoglu, 2017, Neural Discrete Representation Learning, arXiv:1711.00937).

The tradeoff resurfaces wherever an autoencoder feeds a generative model. Latent diffusion tokenisers live on this frontier: raising per-token feature dimension improves reconstruction and makes the generative model's job harder, so the best reconstruction is not the best system (Yao et al., 2025, Reconstruction vs. Generation, arXiv:2501.01423).