One Number, Two Knobs: The ELBO Is a Rate-Distortion Tradeoff

In 2015 a team at Stanford and Google trained a variational autoencoder over sentences. It worked: the decoder produced fluent, grammatical English, and the loss curve descended exactly as it should. The latent variable it was supposed to be conditioning on had become statistically independent of the input (Bowman et al., 2016, Generating Sentences from a Continuous Space, arXiv:1511.06349). The model was a language model with a decorative Gaussian bolted to its side, and nothing in the training objective had registered a complaint.

For a while this was treated as an optimisation pathology with a folklore cure: anneal the KL weight, drop words from the decoder input, wait. It is neither pathological nor curable that way. It is the objective behaving exactly as specified, at a point on a frontier it has no opinion about.

Why this matters: The negative ELBO is the sum of two quantities, a rate and a distortion, that trade off against each other one-for-one. Every configuration on a line of slope minus one scores identically, so a single scalar loss cannot tell you where your model landed. Posterior collapse, the disentanglement disappointment, the modern latent-diffusion tokeniser dilemma and the choice of \(\beta\) are all the same fact viewed from different angles, and none of them is fixable by optimising harder.

TL;DR

• The negative ELBO decomposes exactly as \(D + R\): expected reconstruction cost in nats, plus \(\mathrm{KL}(q(z\mid x)\,\|\,p(z))\). Only the sum enters the loss, so a whole family of models shares one ELBO while differing completely in what the latent encodes (Alemi et al., 2018, Fixing a Broken ELBO, arXiv:1711.00464).
• In the worked example below, a linear-Gaussian VAE moves its rate from 0 to 3.45 nats, about 5 bits, while its ELBO changes by 0.05 nats. The number you minimise moves 3 percent while the representation goes from empty to informative.
• Posterior collapse is the corner at \(R = 0\). With a decoder powerful enough to model \(p(x)\) alone, zero rate is genuinely optimal, not a local minimum (Chen et al., 2017, Variational Lossy Autoencoder, arXiv:1611.02731).
• \(\beta\) is the Lagrange multiplier on a rate budget. Sweeping it traces the frontier; it does not produce better models, only differently positioned ones.
• Rate controls how many bits the code holds and not which bits. Across over 12,000 trained models no unsupervised method reliably disentangled anything, and the paper proves it cannot without inductive bias (Locatello et al., 2019, arXiv:1811.12359).
• Tightening the bound does not resolve the ambiguity and adds a new problem: the importance-weighted bound's encoder gradient signal-to-noise ratio decays as \(O(1/\sqrt{K})\) while the decoder's improves as \(O(\sqrt{K})\) (Rainforth et al., 2018, arXiv:1802.04537).
• The frontier outlived the VAE's moment. Latent diffusion tokenisers sit on it, and the best-reconstructing tokeniser is measurably not the best system (Yao et al., 2025, arXiv:2501.01423).

At a Glance

flowchart LR
  X[Input x] --> E[Encoder q of z given x]
  E -->|"rate R = KL to prior"| Z[Latent z]
  Z --> DEC[Decoder p of x given z]
  DEC -->|"distortion D"| XR[Reconstruction]
  R1["Loss = D + R"] -.-> E
  R1 -.-> DEC
  F["Frontier: any D + R on one line scores the same"] --> R1
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class X,Z blue
  class E,DEC purple
  class XR teal
  class R1,F amber

Two arrows leave the objective, one paid in rate and one in distortion, and the loss adds them. Nothing in the picture says how the total should be split.

[IMAGE: A two-panel scatter plot in the (R, D) plane, rate on the x-axis in nats and distortion on the y-axis. Left panel shows the feasible region as a shaded area bounded below-left by a convex frontier, with a dashed line of slope -1 cutting across it and five model markers sitting on that line. Right panel shows the same five markers with thumbnails of their reconstructions: identical loss, visibly different reconstruction fidelity. Caption: "Every model on the dashed line reports the same ELBO."]

Before the Frontier Had a Name

The variational autoencoder arrived as a solution to an optimisation problem, not a representation-learning one. The reparameterisation trick made the ELBO differentiable with respect to the encoder's parameters, turning a per-datapoint inference procedure into a minibatch training loop (Kingma and Welling, 2014, Auto-Encoding Variational Bayes, arXiv:1312.6114). That the bound also happened to have an autoencoder's shape was a convenient accident, and the field read a great deal into the accident.

Trouble arrived from two directions at once. Practitioners found latents that did nothing. Theorists found the bound could be tightened arbitrarily by importance weighting, which raised the question of what the original bound had been enforcing that the tighter one did not (Burda, Grosse and Salakhutdinov, 2016, Importance Weighted Autoencoders, arXiv:1509.00519).

timeline
    title How the ELBO's ambiguity was found and named
    2013 : Kingma and Welling publish Auto-Encoding Variational Bayes
         : reparameterisation makes the bound trainable at scale
    2015 : IWAE tightens the bound with importance weighting
         : Bowman et al. report a sentence VAE whose latent is ignored
    2016 : Variational Lossy Autoencoder shows an autoregressive decoder makes ignoring the latent optimal
    2017 : beta-VAE adds a knob on the KL term
         : VQ-VAE fixes the rate with a codebook
    2018 : Fixing a Broken ELBO names the rate-distortion frontier
         : Rainforth et al. show tighter bounds degrade the encoder
         : Cremer et al. split the inference gap into approximation and amortisation
    2019 : Locatello et al. prove unsupervised disentanglement needs inductive bias
    2022 : Latent diffusion moves the generative model inside an autoencoder
    2025 : Tokeniser work names the reconstruction-versus-generation dilemma

The 2016 result is the hinge. Chen and colleagues asked what the model was optimising when it collapsed, and answered: a latent costs KL nats to use, so if the decoder can reconstruct as well without it, not using it is strictly cheaper. They then turned that into a design method, restricting the decoder's receptive field so global information could only reach it through \(z\). That is the first appearance of the modern attitude: stop fighting the objective, build the architecture so the point you want is the one the objective prefers.

How the Decomposition Actually Works

Two terms, one loss

For a single data point,

\(D\) is the nats needed to describe \(x\) once you know \(z\): reconstruction error in the currency of code length. \(R\) is the nats needed to describe \(z\) itself relative to the prior, which under a bits-back coding argument is literally the channel capacity the encoder consumes. The loss is their sum, and minimising a sum says nothing about the split. Averaged over the data, the objective's level sets in the \((R, D)\) plane are lines of slope \(-1\), and everything on one such line is, to the optimiser, the same model.

The two corners

The frontier is bounded. At one end sits the auto-decoding limit: \(R = 0\), the posterior equals the prior for every input, the latent carries nothing, and the decoder shoulders the whole distribution alone. Distortion there equals the data entropy \(H\), the best any model does without side information. At the other end sits the auto-encoding limit, distortion near zero and rate as high as the channel will bear. The interesting claim is not that the corners exist but that the region between them is reachable at constant ELBO, which makes the objective genuinely ambiguous rather than merely soft.

Why rate is a bound on information, not information itself

What makes this information theory rather than analogy is a pair of bounds. Writing \(I(X; Z)\) for the mutual information under the joint \(p(x)q(z \mid x)\),

The upper bound is the one people quote and misuse. \(R\) is what the encoder spends; \(I\) is what it conveys. An encoder can burn rate on input-specific jitter that helps reconstruction not at all, and the KL reports it happily. The lower bound is the useful half: if distortion is well below the data entropy, the code demonstrably carries information whatever the rate says.

Hence per-dimension KL is the diagnostic and aggregate KL is not. A 60-dimensional latent reporting 12 nats is healthy at 0.2 nats each and dead at 6 nats in two dimensions and zero in fifty-eight.

Beta as a Lagrange multiplier

Once the frontier is visible, the fix is to choose a point on it rather than accept one. Weighting the KL term,

is the Lagrangian of "minimise distortion subject to a rate budget", with \(\beta\) the multiplier. Large \(\beta\) squeezes the channel toward auto-decoding; small \(\beta\) opens it toward auto-encoding.

The objective is identical to \(\beta\)-VAE, which arrived from the opposite direction. Higgins and colleagues were not thinking about coding; they wanted latent dimensions to correspond to interpretable factors, and found that penalising the KL harder did it (Higgins et al., 2017, beta-VAE, ICLR). The coding reading explains why: with a scarce bit budget the encoder must spend on whichever factors reduce distortion most, which tends to mean large independent factors of variation rather than a smeared code.

It also explains the method's limit, and the limit is severe. Locatello and colleagues trained over 12,000 models across the major methods, metrics and seven datasets, and found that while each method enforces the property its own loss encourages, none reliably produced disentangled representations and none could be selected without supervision. They pair this with a theorem: unsupervised disentanglement is impossible without inductive biases on both model and data. The rate knob decides how many bits the code carries. Nothing in the objective decides which bits.

What actually picks the point

Since the objective abstains, the architecture votes, and the decoder's power dominates. An autoregressive decoder over pixels or tokens is a full density model of \(x\) on its own, so it can hit \(D = H\) at \(R = 0\) and any positive rate is pure cost. A convolutional decoder with a small receptive field cannot assemble global structure alone, so buying it through \(z\) is the cheapest route to low distortion and the solution sits at high rate.

Encoder capacity enters through a different door. Even at a fixed family, the shared encoder will not produce the family's best member for each input, and that shortfall splits into an approximation gap from the family and an amortisation gap from the network (Cremer, Li and Duvenaud, 2018, arXiv:1801.03558). A large amortisation gap gives unusual inputs near-prior posteriors, which looks like collapse on those examples and is routinely misdiagnosed as one.

[IMAGE: Line chart with beta on a log x-axis from 0.01 to 100 and two y-axes, rate in nats on the left and distortion in nats on the right, showing rate falling monotonically and distortion rising monotonically as beta increases, with a shaded band marking the region where reconstructions stay perceptually acceptable. Caption: "Beta does not improve the model; it slides it along a frontier."]

Seeing It in Motion

The frontier is static. The interesting behaviour is how a model finds its point during training, and why the path is often a trap.

flowchart TB
  subgraph WEAK["Weak decoder: limited receptive field"]
    W1[Global structure unavailable locally] --> W2[Latent is cheapest route to low D]
    W2 --> W3[Solution settles at high rate]
  end
  subgraph STRONG["Strong decoder: autoregressive over x"]
    S1[Decoder models p of x alone] --> S2[Any rate is pure added cost]
    S2 --> S3["Solution settles at zero rate"]
  end
  W3 --> OUT["Same objective, opposite outcomes"]
  S3 --> OUT
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  class W1,W2,W3 emerald
  class S1,S2,S3 rose
  class OUT slate

The dynamics add a second trap on top of the static one. Early in training the decoder is bad, so the latent is not yet useful, so the KL term dominates and drives the posterior to the prior. By the time the decoder could exploit a code, the encoder has learned to emit the prior and no gradient points back out. That ordering is what KL annealing addresses, and why it sometimes works and sometimes does not.

stateDiagram-v2
    [*] --> Init
    Init --> KLDominant: decoder still poor, latent unhelpful
    KLDominant --> Collapsed: posterior driven to prior, gradient vanishes
    KLDominant --> Informative: annealing or free bits hold the channel open
    Collapsed --> Collapsed: stable, loss still improving
    Informative --> Informative: rate settles at the budget
    Collapsed --> [*]
    Informative --> [*]

Note the self-loop on Collapsed. It is stable, and the loss keeps improving while the model sits there, because the decoder continues to get better at the unconditional distribution. Nothing in the training curve distinguishes this from health.

[IMAGE: Two training-run traces on shared axes over 100k steps. Top panel plots total loss for both runs, nearly indistinguishable. Bottom panel plots rate in nats: one run rises to a plateau near 8 nats, the other decays to zero by step 5k and stays there. Caption: "The loss curves agree. The rate curves do not."]

Watch It Run

By the Numbers

Sources: IWAE (Burda et al., 2016, arXiv:1509.00519); gradient SNR (Rainforth et al., 2018, arXiv:1802.04537); Flow++ and dequantisation (Ho et al., 2019, arXiv:1902.00275); disentanglement study (Locatello et al., 2019, arXiv:1811.12359); tokeniser figures (Yao et al., 2025, arXiv:2501.01423). The last row is computed in closed form below and involves no measurement. The dequantisation rows are here because they are the clearest published case of a preprocessing choice costing a fixed, measurable slice of a reported likelihood, which is the same category of error as reading an ELBO without its decomposition.

[IMAGE: Grouped bar chart of the four MNIST NLL figures in nats on a truncated axis from 84 to 87, bars for VAE, IWAE K=5 and IWAE K=50, overlaid with a second series in a contrasting colour showing a schematic encoder-gradient SNR falling as K rises. Caption: "The bound tightens left to right; the encoder's gradient quality moves the other way."]

A Concrete Example

Here is the frontier in closed form, small enough to check by hand.

Take one-dimensional data \(x \sim \mathcal{N}(0, 1)\), a one-dimensional latent, prior \(p(z) = \mathcal{N}(0, 1)\), encoder \(q(z \mid x) = \mathcal{N}(x, t)\) and decoder \(p(x \mid z) = \mathcal{N}(z, t)\), with a single free parameter \(t\), the posterior variance. Everything is Gaussian, so both terms are analytic.

Step 1: the rate. For \(q = \mathcal{N}(x, t)\) against \(p = \mathcal{N}(0,1)\), the KL is \(\tfrac{1}{2}(t + x^2 - 1 - \log t)\). Averaging over \(x\) with \(\mathbb{E}[x^2] = 1\):

Step 2: the distortion. \(-\log p(x \mid z) = \tfrac12 \log(2\pi t) + (x - z)^2/(2t)\), and \(\mathbb{E}_q[(x-z)^2] = t\), so the second term is exactly \(\tfrac12\):

Step 3: the sum. The \(\log t\) terms cancel:

That constant is not a coincidence. \(\tfrac12 \log(2\pi e) = 1.4189\) nats is the differential entropy of \(\mathcal{N}(0,1)\), the best any model of this data can do. As \(t \to 0\) the ELBO approaches it exactly.

Step 4: the auto-decoding comparison. Now take the degenerate model that ignores \(z\) entirely: decoder \(\mathcal{N}(0, 1)\), encoder equal to the prior. Rate is zero by construction, and distortion is \(\tfrac12\log(2\pi) + \tfrac12 = 1.4189\). Same number.

Step 5: tabulate.

The last column varies by 0.05 nats, about 3 percent. The rate column spans zero to nearly five bits. A selection procedure that ranks by ELBO, which is what a validation loss does, calls these four models equivalent and picks whichever the seed favoured.

Two details. Distortion goes negative, which is fine: these are differential entropies and a continuous density can exceed one. And the variational gap here is \(O(t^2)\), negligible at these values, so the ELBO really is tracking the marginal likelihood. The ambiguity is not an artefact of a loose bound, and tightening would not help, because there is nothing loose about it.

Where It Breaks

The frontier is not portable between architectures

The achievable \((R, D)\) region depends jointly on encoder capacity, decoder capacity and prior flexibility, and Alemi and colleagues note that the slope-one point's location is a sensitive function of exactly those relative powers. A \(\beta\) tuned on a small convolutional decoder does not transfer to a larger one, and a result reporting "\(\beta = 4\) works best" without the architecture is reporting nothing. Re-sweep after any capacity change.

Annealing treats a symptom that is sometimes not there

KL annealing works when collapse is a transient of the optimisation path: the decoder is weak early, the latent is briefly useless, and the schedule carries the model past that window. It does nothing when \(R = 0\) is the true optimum, because once the schedule ends the gradient points straight back to the corner. Ask first whether your decoder could model the data alone. If it could, the schedule is buying a delay.

Rate is spent, not conveyed

An encoder emitting input-dependent noise pays rate and conveys nothing. It shows up as a healthy-looking aggregate KL with reconstructions no better than a lower-rate model's, common when the encoder's output variance is under-regularised or the latent is much wider than the data warrants. Diagnose by ablation: resample \(z\) from the prior in a subset of dimensions. Dimensions you can resample freely were never carrying anything.

[IMAGE: Horizontal bar chart of per-dimension KL in nats for a 64-dimensional latent, sorted descending, shown twice. Upper chart has three bars above 4 nats and 61 at zero; lower chart has a long gentle slope from 0.4 down to 0.05 nats. Both totals annotated as 12 nats. Caption: "The same aggregate KL, one dead code and one healthy one."]

Free bits fix the shape but can strand capacity

A per-dimension KL floor stops the first few dimensions collapsing and is usually more effective than a global \(\beta\) for that purpose. It also forces every dimension to spend at least the floor whether or not it has anything to say, so a generous floor on a wide latent manufactures exactly the noise-encoding above. Keep the floor small, on the order of a few hundredths of a nat, and check it against a per-dimension histogram rather than setting it once.

Likelihood numbers carry their own bookkeeping

Dequantisation hides a number the same way. Fitting a continuous density to integer data without added noise lets the likelihood diverge; uniform noise fixes that but wastes capacity flattening each quantisation cell, which Flow++ measured at roughly 0.127 bits per dimension on CIFAR-10 against a headline 3.08. Models reported under different dequantisation schemes are not comparable, and neither is comparable to a genuinely discrete likelihood. Bits per dimension is a comparison within a convention, not a property of a model.

Sample quality is a third, independent axis

Even a correctly decomposed, correctly dequantised likelihood says little about whether samples look good: average log-likelihood, Parzen window estimates and visual fidelity are largely independent in high dimensions (Theis et al., 2016, arXiv:1511.01844). The decomposition recovers one hidden dimension of the ELBO; it does not make the ELBO a complete evaluation.

Alternative Designs

The discrete row changed practice most. VQ-VAE replaces the continuous channel with a codebook lookup, capping rate at \(\log K\) nats per position and making the auto-decoding corner unreachable: the decoder cannot ignore a code it is structurally required to read (van den Oord, Vinyals and Kavukcuoglu, 2017, arXiv:1711.00937). It does not abolish the tradeoff, it moves it into the codebook, where it reappears as codebook collapse and an effective rate far below \(\log K\). Finite scalar quantisation attacks exactly that, replacing the learned codebook with a fixed product of small scalar grids and removing the auxiliary losses and the collapse mode with it (Mentzer et al., 2024, arXiv:2309.15505).

How It Is Used in Practice

The largest deployment of this tradeoff today is not a VAE at all. Latent diffusion trains an autoencoder once for perceptual compression, then trains a diffusion model entirely inside its latent space (Rombach et al., 2022, arXiv:2112.10752). The KL weight in that first stage is deliberately tiny, because a full VAE's prior pressure would destroy the spatial structure the diffusion backbone depends on. What is wanted is a mildly regularised code at a chosen rate, and the rate is set by a downsampling factor and a channel count rather than by tuning \(\beta\).

Choosing those two numbers is a rate-distortion decision with a twist. Raising the per-token feature dimension improves reconstruction monotonically and, past a point, makes generation worse, because the generative model now faces a higher-dimensional and harder distribution. Yao and colleagues named this the optimisation dilemma and measured it: rFID and gFID move in opposite directions past the crossover, so the best-reconstructing tokeniser is not the one that yields the best system. Their answer, aligning the tokeniser's latent space to a vision foundation model's features, pushed the frontier outward rather than picking a point on it, reaching 0.28 rFID with 1.35 gFID on ImageNet 256 and 2.11 gFID in 64 epochs, about 21.8 times faster convergence than the original DiT.

That is the 2018 result in new clothing. A single scalar, rFID this time instead of ELBO, fails to determine what you care about, because it sums over a tradeoff whose split matters and which the metric does not report.

Neural compression is the one field where the frontier is the deliverable rather than a nuisance. There \(R\) is the bitstream you are billed for and \(D\) is the quality the user sees, so the curve is the product specification and \(\beta\) is the quality setting. Nobody in compression has ever been confused about what the loss means; the rest of the field borrowed the objective without borrowing that clarity.

[IMAGE: Three-panel comparison of the same photograph decoded from latents at three rates, labelled with their per-image rate in bits and their reconstruction PSNR. Left panel visibly loses small text and fine texture, middle panel is close to indistinguishable, right panel is indistinguishable. Caption: "The frontier is a product decision in compression and an accident everywhere else."]

Insights Worth Remembering

1. The ELBO is not a scalar summary, it is the sum of two coordinates. Reporting it without the rate is reporting a total without saying what was added. Log \(R\) and \(D\) separately from the first run, not after something goes wrong.

2. Posterior collapse is an optimum, not a bug. With a decoder that can model \(p(x)\) alone, zero rate is the global best, and every durable fix is structural: weaken the decoder, floor the KL, make the channel discrete. Optimisation tricks only delay a correct answer you did not want.

3. \(\beta\) moves you along the frontier, never outward. A sweep scans a one-dimensional family. Pushing the frontier itself means changing what the encoder, decoder or prior can represent.

4. Rate upper-bounds information, and the bound can be loose. An encoder can pay for bits it does not use. Trust per-dimension KL histograms and resampling ablations, not an aggregate that merely looks healthy.

5. Tightening the bound changes which network is helped. The importance-weighted bound improves the decoder's gradient signal at \(O(\sqrt{K})\) and degrades the encoder's at \(O(1/\sqrt{K})\). If the representation is the product, tighter is the wrong direction.

6. Every metric that sums across a tradeoff hides the same way. ELBO hides rate against distortion; bits per dimension hides the dequantisation convention; rFID hides what a tokeniser costs the generator. When a number is a sum, ask for the terms.

7. The frontier outlived the VAE as a headline model. It is now the central design question for latent diffusion tokenisers, where the largest generative systems in production spend their compression budget. Nothing in the argument was specific to 2016.

Open Questions

Can the right point on the frontier be chosen without a downstream task? Measured: for a fixed architecture, sweeping \(\beta\) traces the achievable region reliably and cheaply. Unknown: whether any unsupervised criterion identifies the point that serves a downstream task best. Every successful selection reported so far uses downstream performance, so the frontier is traversed by search rather than by principle.

How much of the tokeniser dilemma is rate and how much is geometry? Measured: raising per-token dimension improves reconstruction and, past a crossover, degrades generation, and aligning the latent to foundation-model features improves both at once. Unresolved: whether the generator's difficulty comes from the rate itself or from the shape of the latent distribution at that rate. Alignment helping without reducing dimension suggests geometry carries real weight, but no published decomposition separates the two.

Is there a usable analogue of the decomposition for diffusion? Diffusion objectives are also weighted sums, over noise levels rather than over rate and distortion, and the weighting demonstrably changes what the model prioritises. Whether that yields a frontier with the same clean interpretation is plausible rather than established.

What is the right rate floor for a wide latent? Free bits work, and the floor is chosen empirically in the hundredths of a nat. No principled way to set it from the data's intrinsic dimension exists, and too generous a floor provably manufactures noise-encoding.

Does the ambiguity govern representations learned by other objectives? Contrastive and masked-prediction methods learn codes with no explicit rate term, and their collapse failure modes rhyme with the auto-decoding corner. Whether the mapping is exact has not been established.

Sources and Further Reading

Foundational papers

1. Kingma, D. P., & Welling, M. (2014). "Auto-Encoding Variational Bayes." ICLR. arXiv:1312.6114
2. Burda, Y., Grosse, R., & Salakhutdinov, R. (2016). "Importance Weighted Autoencoders." ICLR. arXiv:1509.00519
3. Alemi, A. A., Poole, B., Fischer, I., Dillon, J. V., Saurous, R. A., & Murphy, K. (2018). "Fixing a Broken ELBO." ICML, PMLR 80:159-168. arXiv:1711.00464
4. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., & Lerchner, A. (2017). "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework." ICLR. OpenReview

Collapse, gaps and what the objective actually enforces

5. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). "Generating Sentences from a Continuous Space." CoNLL. arXiv:1511.06349
6. Chen, X., Kingma, D. P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., & Abbeel, P. (2017). "Variational Lossy Autoencoder." ICLR. arXiv:1611.02731
7. Cremer, C., Li, X., & Duvenaud, D. (2018). "Inference Suboptimality in Variational Autoencoders." ICML. arXiv:1801.03558
8. Rainforth, T., Kosiorek, A. R., Le, T. A., Maddison, C. J., Igl, M., Wood, F., & Teh, Y. W. (2018). "Tighter Variational Bounds are Not Necessarily Better." ICML. arXiv:1802.04537
9. Lucas, J., Tucker, G., Grosse, R. B., & Norouzi, M. (2019). "Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse." NeurIPS. arXiv:1911.02469
10. Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019). "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations." ICML, pp. 4114-4124. Best Paper Award. arXiv:1811.12359

Evaluation and bookkeeping

11. Theis, L., van den Oord, A., & Bethge, M. (2016). "A note on the evaluation of generative models." ICLR. arXiv:1511.01844
12. Ho, J., Chen, X., Srinivas, A., Duan, Y., & Abbeel, P. (2019). "Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design." ICML. arXiv:1902.00275

Discrete bottlenecks and modern tokenisers

13. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). "Neural Discrete Representation Learning." NeurIPS, pp. 6306-6315. arXiv:1711.00937
14. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). "High-Resolution Image Synthesis with Latent Diffusion Models." CVPR. arXiv:2112.10752
15. Mentzer, F., Minnen, D., Agustsson, E., & Tschannen, M. (2024). "Finite Scalar Quantization: VQ-VAE Made Simple." ICLR. arXiv:2309.15505
16. Yao, J., Yang, B., & Wang, X. (2025). "Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models." CVPR (Oral). arXiv:2501.01423

Related library concepts

• The Rate-Distortion View of the ELBO
• Posterior Collapse and the KL Term
• Importance-Weighted Bounds and the SNR Problem
• Amortised Inference and the Amortisation Gap
• Dequantisation and Likelihoods on Discrete Data
• Latent Diffusion and the Autoencoder Bottleneck

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.