The SDE View of Generative Models

Score-based models and diffusion models were developed separately, published separately, and used different notation, different objectives and different samplers. In 2021 they turned out to be the same thing written twice. The framework that showed this (Song et al., 2021, Score-Based Generative Modeling through Stochastic Differential Equations, arXiv:2011.13456) remains the cleanest way to think about the whole family.

Forward and reverse

Write the corruption process as a continuous-time SDE:

with \(f\) the drift, \(g\) the diffusion coefficient, and \(w\) a Wiener process. Anderson's 1982 result gives the time reversal in closed form:

The only unknown is the score of the marginal at each time, \(\nabla_x \log p_t(x)\). Train a network to estimate it with denoising score matching, plug it in, and integrate backwards from noise to data. Generation is a boundary-value problem solved by a numerical integrator.

The two classic choices

\(f(x,t) = -\tfrac{1}{2}\beta(t)x\) with \(g(t) = \sqrt{\beta(t)}\) is the variance-preserving SDE, and its discretisation is exactly DDPM. \(f = 0\) with \(g(t) = \sqrt{\mathrm{d}\sigma^2(t)/\mathrm{d}t}\) is variance-exploding, and its discretisation is the noise-conditional score network. Two literatures, two SDEs, one framework. The choice of \(f\) and \(g\) is the noise schedule expressed in continuous time.

The probability flow ODE

For every diffusion SDE there is a deterministic ODE with identical marginal distributions at every \(t\):

Same marginals, no randomness. Three consequences follow, and they are the practical payoff of the whole framework.

Deterministic sampling becomes available, which is what DDIM is. The map from noise to data is a bijection, so latents can be inverted and interpolated. And because the ODE is a continuous normalising flow, the exact log-likelihood is computable by integrating the instantaneous change of variables, which gives diffusion models a likelihood they were not designed to have.

The ODE and the SDE trade differently. The SDE's noise repeatedly corrects accumulated error, so it is better at many steps; the ODE is smooth and integrable with high-order solvers, so it is far better at few steps.

What the unification bought

Concretely: any sampler developed for one family works for the other; the noise schedule became a continuous design choice rather than a discrete sequence; exact likelihood evaluation became possible; predictor-corrector sampling emerged by combining a numerical SDE step with Langevin correction steps; and controllable generation reduced to adding a conditional term to the score, which is the mathematical basis of both classifier and classifier-free guidance.

When it breaks

Continuous time is a description, not an implementation. Everything is ultimately discretised, and the discretisation is where quality is won or lost. The framework tells you what you are approximating; it does not make the approximation good.

The reverse SDE assumes an exact score. The learned score is wrong in low-density regions, and errors compound over the trajectory. The theory is exact and the model in it is not, which is why sampling remains an engineering problem after the mathematics is settled.

Exact likelihood is expensive and rarely reported honestly. Each evaluation requires integrating the ODE plus a divergence estimate, usually via Hutchinson's stochastic trace estimator, so the result is itself stochastic. Likelihoods quoted for diffusion models are estimates whose variance depends on the number of solver steps and trace samples used.

Guidance leaves the framework. Classifier-free guidance modifies the score by extrapolation, and the modified field is not the score of any normalised distribution. The reverse SDE's correctness and the ODE's likelihood both assume the true score, so guided sampling has no guarantees from this theory at all, despite being what every deployed system does.