Dequantisation and Likelihoods on Discrete Data

An 8-bit image is 3,072 integers in \(\{0, \dots, 255\}\). A normalising flow or a Gaussian mixture is a density on \(\mathbb{R}^{3072}\). Fit the second to the first directly and the likelihood is unbounded above: the model puts ever-narrower spikes on the 256 lattice values, each spike's density growing without limit, and the reported score improves forever while the model learns nothing about images. Any bits-per-dimension figure produced this way is not comparable to anything.

The standard repair is to add noise. Uniform dequantisation replaces each integer \(x\) with \(y = x + u\) where \(u \sim \mathcal{U}[0, 1)^D\), and fits the continuous model to \(y\) (Uria, Murray and Larochelle, 2013, RNADE, arXiv:1306.0186). Spikes no longer pay, because probability mass must now cover a unit hypercube around each lattice point.

Why it is a lower bound, not a fudge

The fix is principled. Define the discrete model implied by the continuous one as the integral of \(p_{\text{model}}\) over the unit cube at each lattice point, \(P(x) = \int_{[0,1)^D} p_{\text{model}}(x + u)\, du\). Then by Jensen,

The dequantised objective you are optimising is a lower bound on the log-likelihood of the discrete data under the induced discrete model (Theis, van den Oord and Bethge, 2016, A note on the evaluation of generative models, arXiv:1511.01844). Reporting it is honest, comparisons between models that use the same dequantisation are valid, and the bound is loose by exactly the amount the model wastes modelling the noise.

Variational dequantisation and what uniform noise costs

Uniform noise is one choice of a distribution \(q(u \mid x)\), and it is a poor one: it asks the model to be flat inside every hypercube, which no smooth density is. Replace it with a learned conditional \(q_\phi(u \mid x)\) supported on \([0,1)^D\) and the bound becomes

with uniform \(q\) recovered as the special case where the entropy term is constant. This is variational dequantisation, and the gap it closes is measurable rather than theoretical: on CIFAR-10, switching Flow++ from variational back to uniform dequantisation costs about 0.127 bits per dimension, against a headline figure of 3.08 (Ho et al., 2019, Flow++, arXiv:1902.00275). On a 3,072-dimensional image that is roughly 390 bits per picture given away to a preprocessing decision.

When it breaks

Bits per dimension is only comparable within a dequantisation scheme. A model using variational dequantisation and one using uniform noise are not on the same scale, and neither is comparable to a model with a genuinely discrete likelihood, which needs no bound at all. Check what a leaderboard is measuring before trusting a 0.05 bpd difference.

The noise scale must match the data's quantisation. Data stored as 8-bit but actually derived from a 5-bit source has an effective step of 8, and dequantising with unit noise leaves the spike problem intact at coarser spacing. Audio resampled or images passed through a lossy codec both hit this.

Dequantisation cannot rescue ordinal-free categories. Adding continuous noise to a token index asserts that index 7 lies between 6 and 8, which is false for vocabulary IDs. Categorical data needs argmax flows, multinomial diffusion, or an honestly discrete model, not a smoothing hack.

The bound is tighter than it looks for a bad model and looser for a good one. A weak model's predictions are already smooth inside each cube, so uniform noise costs it little; a sharp model pays the most. The dequantisation penalty therefore grows as the rest of your model improves, which makes it easy to under-weight early in a project and expensive later.

It does not fix the deeper problem. A good likelihood and good samples are largely independent in high dimensions, so a carefully dequantised, well-calibrated bits-per-dimension figure still tells you little about whether the samples are any good (Theis et al., 2016). Dequantisation makes the number meaningful, not sufficient.