Score Matching and Its Denoising Form
How targeting the gradient of log density eliminates the partition function, why the naive form requires an intractable Hessian trace, and how adding noise makes the objective a simple regression.
The partition function blocks maximum likelihood for energy-based models. Score matching removes it with one observation: the quantity \(\nabla_x \log p(x)\), called the score, does not depend on the normalising constant at all, because \(Z\) is constant in \(x\) and vanishes on differentiation. Fit the score instead of the density, and the intractable integral never appears.
The naive objective and its obstacle
The natural target is
which cannot be evaluated because the data score is unknown. Hyvarinen showed by integration by parts that this is equivalent, up to a constant, to
which contains only quantities computable from the model and the data (Hyvarinen, 2005, Estimation of Non-Normalized Statistical Models by Score Matching, JMLR 6:695-709).
This is a genuine result and computationally hopeless at scale. The trace of the Jacobian of \(s_\theta\) requires \(d\) separate backward passes for \(d\) dimensions, so one training step on a 3,072-dimensional image costs 3,072 backward passes. Sliced score matching reduces this by projecting onto random directions, which helps but adds variance.
The denoising form
Denoising score matching (Vincent, 2011, A Connection Between Score Matching and Denoising Autoencoders, Neural Computation 23(7)) changes the target distribution. Instead of matching the score of the data, match the score of the data convolved with a Gaussian of width \(\sigma\). For that perturbed distribution the conditional score has a closed form:
and the objective becomes a plain regression:
Sample a data point, add Gaussian noise, and train the network to predict the direction back toward the clean point, scaled by \(1/\sigma^2\). No Hessian, no trace, no partition function, one forward and one backward pass. This is the entire diffusion training objective, arrived at from the energy-based side rather than the variational side, and noticing that these were the same objective was the step that unified the two literatures.
Why one noise level is not enough
Fitting the score at a single \(\sigma\) gives a model that is accurate only near the data manifold, since that is where the perturbed distribution has mass and therefore where training examples land. Sampling from such a model requires starting far from the manifold, exactly where the score estimate is meaningless.
Noise-conditional score networks (Song and Ermon, 2019, arXiv:1907.05600) train one network across a geometric ladder of noise levels, conditioning on \(\sigma\). Large \(\sigma\) smooths the distribution enough that its score is informative everywhere and its modes are connected; small \(\sigma\) preserves detail. Sampling anneals from large to small. That ladder is the noise schedule of a diffusion model under a different name.
When it breaks
The estimate is worst in low-density regions and that is where sampling starts. This is the manifold hypothesis biting: real data occupies a thin set, gradient information away from it is scarce, and no amount of training on data-adjacent noise fixes a region no training sample reaches. Multiple noise scales are the fix, and they work because they change the training distribution, not because they improve the estimator.
Loss weighting across noise levels is a design decision, not a detail. The natural scaling \(1/\sigma^2\) makes small-\(\sigma\) terms dominate the gradient by orders of magnitude. Reparameterising the network to predict the noise, or weighting the loss to equalise contributions, is required for the ladder to train at all, and the choice of weighting materially changes the resulting model.
Score matching is blind to relative mode mass. The score is a local gradient, so a model can have every mode in the right place with badly wrong relative probabilities and still fit the score well in each region. If two modes are separated by a region of near-zero density, no local training signal connects them. This is why diffusion sampling relies on the large-noise levels to bridge modes, and why a truncated noise ladder produces samples whose class balance is subtly wrong.
Exact likelihood requires going back through the ODE. Having sidestepped \(Z\) during training, you have not obtained it. Computing a likelihood from a score model means integrating the probability flow ODE together with a divergence estimate, which is expensive and approximate.
12 flashcards for this concept
Click a card to reveal the answer.