Normalising Flows and the Cost of Invertibility
How change of variables turns a simple density into a complex one with an exact likelihood, why the Jacobian determinant constrains every architectural choice, and what that constraint costs relative to VAEs and diffusion.
Most generative models give up exact likelihoods. VAEs optimise a lower bound; GANs have no likelihood at all; diffusion models have one in principle and it is expensive to evaluate. Normalising flows give you the exact log-density of any point, in one forward pass, with no bound and no approximation. The price is that every layer must be invertible and have a tractable Jacobian determinant, and that constraint reaches into every design decision.
Change of variables
If \(z \sim p_Z\) and \(x = f(z)\) for an invertible, differentiable \(f\), then
The first term is the density of the simple base distribution, usually a standard Gaussian, evaluated at the point's preimage. The second corrects for how much the transformation stretched or compressed volume there. Compose many such transforms and the log-determinants add, so a deep flow is a sequence of volume-tracked warps from a Gaussian to the data distribution.
Training is direct maximum likelihood: push data backwards through the inverse, evaluate the base density, add the log-determinants, and maximise. There is no adversarial game, no bound, no sampling in the training loop. Sampling is the forward direction, and it is a single pass.
Why the Jacobian dictates the architecture
For general \(f\) over \(d\) dimensions, computing \(\det \partial f / \partial z\) costs \(O(d^3)\), which is hopeless for images. Every flow architecture is an answer to the question of how to be expressive while keeping that determinant cheap.
Coupling layers, used in RealNVP and Glow, split the input in half, leave the first half unchanged, and transform the second half by a scale and shift predicted from the first. The Jacobian is triangular, so its determinant is the product of the diagonal, which is just the product of the predicted scales. The transform network can be arbitrarily complex because it is never inverted, only evaluated. The cost is that half the dimensions pass through untouched per layer, so many layers with alternating masks are needed.
Autoregressive flows make each output depend on all previous inputs, again giving a triangular Jacobian. Masked autoregressive flow is fast to evaluate density and slow to sample, since sampling is sequential over dimensions; inverse autoregressive flow reverses both properties. Which one you want depends on whether the flow is being used for density estimation or as a variational posterior, and the asymmetry is a genuine, unavoidable feature of the construction.
Continuous flows define the transform as the solution of an ODE, replacing the log-determinant with an integral of the trace of the Jacobian, which can be estimated cheaply with a Hutchinson estimator. This buys freedom in the network at the cost of numerical integration in the training loop.
When it breaks
Dimension preservation is not optional. An invertible map cannot change dimensionality, so the latent space has exactly as many dimensions as the data. For a 256x256 image that is 196,608 latent dimensions, all of which must be modelled, even though the data manifold is vastly lower dimensional. Flows cannot compress, which is precisely the thing latent diffusion exploits to get cheap.
Expressiveness per parameter is poor. Because each layer is constrained, flows need many more layers and parameters than an unconstrained network to reach comparable quality. Glow used around 200 million parameters for 256x256 faces at a sample quality that later models matched with far less. The constraint is a real tax, not an implementation detail.
The manifold mismatch causes pathologies. Fitting a full-dimensional density to data concentrated on a lower-dimensional manifold pushes the model toward assigning enormous density to a thin region, which shows up as numerical instability and as the well-known result that flows trained on one image dataset can assign higher likelihood to out-of-distribution images from another (Nalisnick et al., 2019, Do Deep Generative Models Know What They Don't Know?, arXiv:1810.09136). Exact likelihood is not the same as a good detector of what belongs.
Where flows still win. Anywhere the exact density is the product rather than the samples: variational posteriors that need to be flexible and evaluable, importance sampling proposals, physics applications such as lattice field theory and Boltzmann generators where the target density is known and unbiased sampling is the point, and low-dimensional density estimation where the dimension-preservation constraint costs nothing. The insight that survived into diffusion and flow matching is the continuous-time view, which kept the change-of-variables machinery and dropped the requirement that every layer be separately invertible.
14 flashcards for this concept
Click a card to reveal the answer.