Variational Inference and the ELBO

MCMC is asymptotically exact and slow. Variational inference is fast and wrong in a characterisable way, and for a large class of problems that is the better trade. It replaces the integration problem with an optimisation problem: choose a family of tractable distributions \(\mathcal{Q}\), and find the member closest to the posterior.

Closest in which sense is the decision that determines everything else. The standard choice is reverse KL:

Why the objective becomes computable

That KL contains the intractable posterior, so it cannot be evaluated. Expanding it gives

The last term is the log evidence, which is constant in \(q\). Dropping it and flipping the sign gives the evidence lower bound:

Maximising the ELBO is exactly minimising the KL to the posterior, and every term is computable. The two pieces read as an accuracy term, how well \(q\) explains the data, and a complexity term, how far \(q\) has moved from the prior. Since \(\log p(x) = \mathrm{ELBO}(q) + \mathrm{KL}(q \| p(\theta\mid x)) \geq \mathrm{ELBO}(q)\), the bound's gap is precisely the approximation error, which is why it is a lower bound on the evidence.

This is the same object that trains a variational autoencoder, where \(q\) is an encoder network producing per-datapoint parameters. Amortisation, learning one network that maps inputs to variational parameters instead of optimising parameters per datapoint, is what scales the idea to large datasets.

What reverse KL does to the answer

\(\mathrm{KL}(q \| p) = \int q \log(q/p)\) is large wherever \(q\) has mass and \(p\) does not, and contributes nothing wherever \(p\) has mass and \(q\) does not. The optimiser therefore avoids placing mass where the posterior is thin, and pays no penalty for ignoring regions the posterior occupies. The consequences are systematic and predictable:

Variance is underestimated. \(q\) fits inside the posterior rather than covering it. Credible intervals from a variational fit are typically too narrow, sometimes dramatically, and this is the single most important caveat when the output is used for decisions.

Multimodal posteriors get one mode. Covering two modes would require mass in the low-density valley between them, which reverse KL penalises. Forward KL, \(\mathrm{KL}(p\|q)\), is mass-covering and would spread across both, and it is not used because it requires expectations under the intractable posterior.

Mean-field factorisation destroys correlations. The common choice \(q(\theta) = \prod_j q_j(\theta_j)\) makes coordinates independent by assumption. If the posterior has correlated parameters, the fitted factorised \(q\) shrinks to fit inside the correlated ellipse, compounding the variance underestimate. Full-rank Gaussian \(q\) and normalising-flow families both relax this at higher cost.

Making it generic

Classical VI derived coordinate-ascent updates per model, which is why it was a specialist tool. Two ideas made it automatic. The reparameterisation trick writes \(\theta = \mu + \sigma \odot \epsilon\) with \(\epsilon \sim \mathcal{N}(0,I)\), moving the randomness outside the parameters so the gradient of an expectation becomes an expectation of a gradient, with far lower variance than the score-function estimator. Stochastic optimisation then estimates the ELBO gradient from minibatches. Automatic differentiation variational inference packages both, so a model written once can be fitted by NUTS or by VI without rewriting it (Kucukelbir et al., 2017, Automatic Differentiation Variational Inference, JMLR 18).

When it breaks

The ELBO cannot tell you how wrong the answer is. Its gap to the log evidence is the KL you cannot compute. A converged ELBO means the optimiser finished, not that \(q\) resembles the posterior. Comparing ELBOs across different variational families is also invalid as a measure of posterior quality, since a richer family raises the bound whether or not it improved the fit in any way you care about.

Point estimates are more reliable than uncertainties. The posterior mean from VI is often close to the truth while the posterior variance is badly underestimated. Using VI for a MAP-like summary is defensible; using it for a calibrated interval usually is not, without checking against MCMC on a subsample.

Local optima are real. The ELBO is non-convex in general, so initialisation matters and different runs can land on different modes, especially with mixture models where component-permutation symmetry creates many equivalent optima.

Mean-field is the default and is rarely appropriate. Most posteriors of interest have correlated parameters, particularly in hierarchical models. Reaching for full-rank or a flow-based family should be the considered choice, not an advanced option.