The Score Is All You Need: How Energy-Based Models, Langevin Dynamics and Diffusion Became One Theory

In 2019 an energy-based model trained with 60 Langevin steps per negative sample, a 10,000-image replay buffer and spectral normalisation on every layer reached an FID of 40.58 on CIFAR-10 (Du & Mordatch, 2019, arXiv:1903.08689). Three years later a network that never computed an energy, never ran a Markov chain in training and never touched a normalising constant reached 1.97 in 35 network evaluations (Karras et al., 2022, arXiv:2206.00364). The second model is not a different species. It learns exactly the quantity the first one differentiated in order to sample: the gradient of the log density, the score.

A density \(p(x) = e^{-E(x)}/Z\) is hard because \(Z\) integrates over every possible image; its gradient \(\nabla_x \log p(x) = -\nabla_x E(x)\) contains no \(Z\). Learn that gradient from samples and score matching, Langevin sampling, Tweedie's formula, DDPM and the probability-flow ODE become views of one object.

Why this matters: Every production image, video, audio and weather diffusion model is a score estimator plus a numerical integrator. Knowing which parts are fixed by theory (the score, the reverse SDE, Tweedie's identity) and which are free choices (schedule, parameterisation, loss weighting, solver) lets you read any new diffusion paper as a point on a known map, debug samplers instead of guessing, and see why energy-based models still matter for composition.

TL;DR

• The partition function was sidestepped, not solved. Because \(\nabla_x \log Z = 0\), a model that needs only the score never pays for normalisation. Hyvärinen's 2005 score matching made that trainable, but its Jacobian trace naively costs one backward pass per input dimension: 3,072 for a 32×32 RGB image.
• Denoising is score estimation. Vincent (2011) proved denoiser training equals score matching on the noised density, and Tweedie's formula makes it exact: \(\mathbb{E}[x \mid \tilde{x}] = \tilde{x} + \sigma^2 \nabla \log p_\sigma(\tilde{x})\).
• Noise makes sampling possible. Langevin dynamics with exact scores misweights separated modes; NCSN's 10 noise levels (1,000 evaluations) fixed it and reached Inception score 8.87 on CIFAR-10.
• The SDE view closed the loop. DDPM's noise prediction (FID 3.17, 1,000 steps) is a rescaled score; both it and NCSN discretise one reverse-time SDE, and an ODE with the same marginals gives exact likelihoods and cut solver evaluations by over 90%.
• After 2021 the gains were design on fixed theory. EDM moved a pretrained model from FID 3.01 to 1.97 at 35 evaluations by changing preconditioning, weighting and solver. One Tweedie step alone cut NCSNv2's FID from 31.75 to 10.87.

At a Glance

flowchart LR
    D["Data samples"] --> N["Add Gaussian noise"]
    N --> T["Train denoiser by regression"]
    T --> S["Score of noised density"]
    S --> R["Reverse SDE or ODE solver"]
    R --> G["Generated samples"]
    E["Energy model"] -.->|"minus gradient"| S
    S -.->|"Tweedie identity"| X["Posterior mean"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class D,N blue
    class T,S purple
    class R,X teal
    class G emerald
    class E slate

The solid path is how modern diffusion models are built. The dashed edges make it one theory: an energy's negative gradient is a score, and a score is a denoiser.

Before the Score: Fighting the Partition Function

Pick any scalar \(E_\theta(x)\), set \(p_\theta(x) \propto e^{-E_\theta(x)}\), and you have a density with no invertibility constraint or causal ordering. The price is \(Z_\theta = \int e^{-E_\theta(x)}dx\). Maximum likelihood still has a clean gradient,

but the second term is an expectation under the model, so every update needs fresh model samples. Hinton's contrastive divergence approximated them with a few MCMC steps started at the data (Hinton, 2002, Neural Computation 14(8)). That suited restricted Boltzmann machines and scaled badly to deep energies on images, where chains must cross wide low-density regions.

[IMAGE: Two panels. Left: a 2-D energy landscape with two wells and a high ridge, with a short MCMC chain stuck in the left well and "Z" written above. Right: the gradient field of the same landscape as arrows, with "Z" struck through. Caption: "The density needs Z; its gradient does not. Every method in this article exploits that asymmetry."]

Du and Mordatch pushed MCMC-trained EBMs near their limit with replay-buffer Langevin chains, spectral normalisation and L2 penalties on energies, whose values otherwise "would fluctuate to numerically unstable values"; their ImageNet 32×32 model trained for five days on 32 GPUs. Nijkamp et al. then showed an uncomfortable fact: EBMs trained with short-run, non-convergent Langevin chains generate realistic images although the chains never reach the model's stationary distribution (Nijkamp et al., 2019, NeurIPS, arXiv:1904.09770), and learning "can be effective and stable even when MCMC samples have much higher energy than true steady-state samples" (Nijkamp et al., 2020, AAAI, arXiv:1903.12370). These models learned a good sampler, not necessarily a faithful energy.

A separate thread had been growing for fifteen years without MCMC in training.

timeline
    title From partition functions to one score-based theory
    1956 : Robbins derives what Efron later popularises as Tweedie's formula
    1982 : Anderson shows reverse-time diffusions need only the score
    2002 : Hinton introduces contrastive divergence
    2005 : Hyvarinen proposes score matching
    2011 : Vincent links denoising autoencoders to score matching
    2015 : Sohl-Dickstein et al. introduce diffusion probabilistic models
    2019 : Song and Ermon release NCSN with annealed Langevin dynamics
         : Du and Mordatch scale MCMC-trained EBMs
    2020 : Ho et al. DDPM reaches FID 3.17
         : Song et al. post the SDE framework, published at ICLR 2021
    2022 : Karras et al. EDM separates the design space

Hyvärinen's paper is the pivot: match gradients of densities instead of densities (Hyvärinen, 2005, JMLR 6). Sohl-Dickstein et al. independently built the "destroy structure slowly, then learn to restore it" model from nonequilibrium thermodynamics (Sohl-Dickstein et al., 2015, arXiv:1503.03585).

How Score-Based Modelling Actually Works

Score matching: learning a gradient you cannot observe

For an energy model the score is \(s_\theta(x) = -\nabla_x E_\theta(x)\). The natural objective is the Fisher divergence \(\tfrac{1}{2}\mathbb{E}_{p_{\text{data}}}\lVert s_\theta(x) - \nabla_x \log p_{\text{data}}(x)\rVert^2\), which looks useless because the data score is unknown. Hyvärinen integrated by parts to show that, up to a constant and under mild boundary conditions, it equals

Everything depends only on the model. The norm term pulls scores to zero at data points, so data sits at local maxima; the trace term rewards negative curvature there, so those maxima are sharp. The trace is the cost: exact computation takes \(D\) backward passes. Sliced score matching replaced it with random projections needing only Hessian-vector products (Song et al., 2019, UAI, arXiv:1905.07088), but a different escape won.

Denoising score matching: the trace disappears

Vincent changed the target (Vincent, 2011, Neural Computation 23(7)). Corrupt a clean sample, \(\tilde{x} = x + \sigma z\) with \(z \sim \mathcal{N}(0, I)\), giving the noised marginal \(p_\sigma(\tilde{x}) = \int p_{\text{data}}(x)\,\mathcal{N}(\tilde{x}; x, \sigma^2 I)\,dx\). The corruption kernel's score is known: \(\nabla_{\tilde{x}} \log q_\sigma(\tilde{x} \mid x) = -(\tilde{x} - x)/\sigma^2\). Regressing onto it,

has the same minimiser as explicit score matching on \(p_\sigma\). The conditional score is a noisy but unbiased label: averaged over every clean \(x\) that could have produced \(\tilde{x}\), it equals \(\nabla \log p_\sigma(\tilde{x})\). No second derivatives, no MCMC, only least squares, but for the noised distribution, which approximates the data only at small \(\sigma\).

Tweedie's formula: the denoiser and the score are one function

That averaging argument is Robbins' 1956 empirical Bayes result, popularised by Efron as Tweedie's formula (Efron, 2011, JASA 106(496)). For Gaussian corruption,

Read it both ways. A score gives you the minimum-mean-squared-error denoiser for free. An optimal L2 denoiser \(D(\tilde{x}; \sigma)\) gives you the score, \((D(\tilde{x};\sigma) - \tilde{x})/\sigma^2\). Kadkhodaie and Simoncelli used it (crediting Miyasawa, 1961) to sample from the prior implicit in a blind denoiser with no generative training (Kadkhodaie & Simoncelli, 2021, NeurIPS, arXiv:2007.13640). It is why EDM can train a denoiser and call it a score model.

[IMAGE: A 1-D bimodal density (dark blue), its Gaussian-blurred version (light blue), and at one point x-tilde an arrow of length sigma squared times the score, landing on the posterior mean. Caption: "Tweedie's formula: step along the noised score by sigma squared and you land on the average clean signal consistent with the observation."]

Langevin dynamics and why it needs annealing

Langevin dynamics turns a score into a sampler:

Drift climbs the log density; noise stops collapse onto a mode. As \(\epsilon \to 0\) and steps grow, the chain converges to \(p\) under regularity conditions, and nothing in the update needs \(Z\). Song and Ermon identified two failures on images (Song & Ermon, 2019, arXiv:1907.05600). First, images lie near a low-dimensional manifold, so score matching gives no signal in the empty regions where noise-initialised chains start. Second, "when two modes of the data distribution are separated by low density regions, Langevin dynamics will not be able to correctly recover the relative weights of these two modes in reasonable time." Their experiment used exact scores: the fault is the score's blindness to mode weights, not the network.

The fix was to perturb data at a geometric sequence \(\sigma_1 > \dots > \sigma_L\) and train one Noise Conditional Score Network \(s_\theta(x, \sigma)\) on all levels. Large \(\sigma\) fills space so the score is defined everywhere and mode weights leak into it; small \(\sigma\) keeps the result close to the data. NCSN used \(L = 10\) levels from \(\sigma_1 = 1\) to \(0.01\) and \(T = 100\) Langevin steps per level, 1,000 score evaluations per image, for Inception score 8.87 and FID 25.32. NCSNv2 set the levels from theory (Song & Ermon, 2020, arXiv:2006.09011): \(\sigma_1\) should be "as large as the maximum Euclidean distance between all pairs of training data points." CIFAR-10's median pairwise distance is about 18, so \(\sigma_1 = 1\) was far too small; they tested 50.

DDPM: the same model, predicting noise

Ho, Jain and Abbeel came from the Sohl-Dickstein lineage (Ho et al., 2020, arXiv:2006.11239). Variances rise linearly from \(\beta_1 = 10^{-4}\) to \(\beta_T = 0.02\) over \(T = 1000\) steps, so

A network \(\epsilon_\theta(x_t, t)\) predicts the noise with plain mean-squared error. The authors state this parameterisation "reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling." Applying the DSM target to the scaled kernel gives the dictionary

The unweighted loss gave FID 3.17; the variational-bound weighting gave better likelihood but FID 13.51.

The SDE framework: one equation behind both

Song et al. took the number of noise levels to infinity (Song et al., 2021, ICLR, arXiv:2011.13456). Noising becomes an Itô SDE \(dx = f(x, t)\,dt + g(t)\,dw\). NCSN is the variance-exploding choice \(f = 0\), \(g = \sqrt{d\sigma^2/dt}\); DDPM is variance-preserving, \(f = -\tfrac{1}{2}\beta(t)x\), \(g = \sqrt{\beta(t)}\). Anderson proved the time reversal of a diffusion is a diffusion (Anderson, 1982, Stochastic Processes and their Applications 12(3)):

The only unknown is the time-dependent score, which a noise-conditional network learns. Annealed Langevin and DDPM ancestral sampling are two discretisations of this equation. The paper added a predictor-corrector sampler (a reverse-SDE step, then Langevin corrections at the new level) and a deterministic process with identical marginals, the probability-flow ODE:

The factor of one half is the entire difference: the SDE's extra half-score is balanced by its noise. The ODE is a continuous normalising flow, so it gives exact log-likelihoods, invertible latents and adaptive solvers; a looser tolerance cut evaluations "by over 90% without affecting the visual quality of samples." The best model reached FID 2.20 on CIFAR-10 and 2.99 bits/dim with a likelihood-oriented SDE.

EDM: separating theory from design choices

By 2022 each named method bundled a schedule, parameterisation, loss weighting and sampler; Karras et al. unbundled them. Take \(x = y + n\) with \(n \sim \mathcal{N}(0, \sigma^2 I)\) and \(\sigma(t) = t\). Tweedie turns the probability-flow ODE into

so sampling moves along the line from the denoised estimate to the current point. The raw network \(F_\theta\) is preconditioned so inputs and targets have unit variance at every level:

with \(c_{\text{skip}} = \sigma_{\text{data}}^2/(\sigma^2 + \sigma_{\text{data}}^2)\), \(c_{\text{out}} = \sigma\sigma_{\text{data}}/\sqrt{\sigma^2 + \sigma_{\text{data}}^2}\), \(c_{\text{in}} = 1/\sqrt{\sigma^2 + \sigma_{\text{data}}^2}\) and \(\sigma_{\text{data}} = 0.5\). At low noise the network predicts a small correction; at high noise it predicts the clean image. Predicting \(\epsilon\) at high noise, as DDPM does, multiplies network error by \(\sigma\) when converting back to an image; this interpolation avoids that.

Training samples \(\ln\sigma \sim \mathcal{N}(-1.2, 1.2^2)\), concentrating on mid-range noise where the target is neither trivial nor hopeless, with weight \(\lambda(\sigma) = (\sigma^2 + \sigma_{\text{data}}^2)/(\sigma\sigma_{\text{data}})^2\). Sampling uses

which spends steps at low noise where trajectories curve, integrated with second-order Heun. Optional "churn" re-injects noise per step; with their improved CIFAR-10 training "any amount of stochastic sampling was detrimental," while a retrained ImageNet-64 model still benefited and reached FID 1.36. The changes took a pretrained VP model from 3.01 to 1.97 at 35 evaluations and lifted an existing ImageNet-64 model from 2.07 to 1.55 with sampler changes alone. No new theory was needed.

Seeing It in Motion

Every model above fits one pipeline; the middle outputs are coordinates for one learned object.

flowchart TB
    subgraph Train["Training, no MCMC"]
        X0["Clean sample"] --> XN["Noised input at sigma"]
        XN --> PRE["Preconditioned denoiser D"]
        PRE --> LOSS["Weighted L2 loss"]
    end
    subgraph Views["Equivalent outputs"]
        PRE --> SC["Score via Tweedie"]
        PRE --> EPS["Noise prediction, DDPM"]
        PRE --> EN["Energy gradient, EBM"]
    end
    subgraph Sample["Sampling"]
        SC --> SDE["Reverse SDE"]
        SC --> ODE["Probability-flow ODE"]
        SDE --> OUT["Sample"]
        ODE --> OUT
        ODE --> LIK["Exact likelihood"]
    end

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class X0,XN blue
    class PRE,SDE,ODE purple
    class LOSS amber
    class SC,EPS teal
    class EN slate
    class OUT,LIK emerald

What does one step of a modern deterministic sampler do? Here is one EDM Heun step.

sequenceDiagram
    participant S as Sampler
    participant D as Denoiser
    S->>D: x at current sigma
    D-->>S: denoised estimate
    S->>S: slope is x minus estimate, over sigma
    S->>S: Euler proposal at next sigma
    S->>D: proposal at next sigma
    D-->>S: second denoised estimate
    S->>S: average the two slopes, update x
    Note over S,D: Two network calls per step
    Note over S,D: Last step to sigma zero is Euler only

[IMAGE: Line plot of FID against network evaluations (log scale, 8 to 1024) for one pretrained unconditional CIFAR-10 VP model under the original Euler sampler, Heun with EDM steps, and adaptive RK45, with Heun plateauing near FID 2 around 35 evaluations. Caption: "Same network, different integrators. Most sampler speedup since 2020 is numerical analysis, not learning."]

By the Numbers

Sources: Du & Mordatch, 2019; Song & Ermon, 2019; Song & Ermon, 2020; Ho et al., 2020; Song et al., 2021; ScoreFlow (Song, Durkan, Murray & Ermon, 2021, arXiv:2101.09258); DPM-Solver (Lu et al., 2022, arXiv:2206.00927); EDM Table 2 (Karras et al., 2022); Song, Dhariwal, Chen & Sutskever, 2023, arXiv:2303.01469. Unconditional unless marked. The Score SDE evaluation count is an estimate from the paper's PC1000 configuration (1,000 predictor plus 1,000 corrector steps), not a reported figure.

Two eras are visible. From 2019 to 2021 FID fell about tenfold at roughly 1,000 evaluations: better scores. From 2021 to 2023 evaluations fell two to three orders of magnitude at similar quality: better integration, possible only once sampling was recognised as solving an ODE.

[IMAGE: Log-log scatter of CIFAR-10 FID against evaluations per sample for NCSN, NCSNv2, DDPM, Score SDE, DPM-Solver-10, DPM-Solver-20, EDM and one-step consistency, coloured purple where the gain came from training and teal where it came from sampling. Caption: "First the score got better, then the integrator did."]

A Concrete Example

All of this can be checked by hand in one dimension. Data: weight 0.3 on \(\mathcal{N}(-2, 0.5^2)\) and 0.7 on \(\mathcal{N}(2, 0.5^2)\). Noise at \(\sigma = 1\); convolving Gaussians adds variances, so each noised component has \(v = 0.25 + 1 = 1.25\). We compute the score at \(\tilde{x} = 0.5\), take a Langevin step, and denoise.

Step 1: weighted component terms. Each is \(w_k \exp(-(\tilde{x}-\mu_k)^2/2v)\); the shared factor \(1/\sqrt{2\pi v}\) will cancel.

• Left: \((2.5)^2/2.5 = 2.5\), \(e^{-2.5} = 0.0821\), times 0.3 gives \(0.02463\).
• Right: \((1.5)^2/2.5 = 0.9\), \(e^{-0.9} = 0.4066\), times 0.7 gives \(0.28460\).

Step 2: responsibilities. \(r_{\text{left}} = 0.02463/(0.02463 + 0.28460) = 0.0796\), \(r_{\text{right}} = 0.9204\). The only normalisation is a sum over two components, not an integral over the real line.

Step 3: the score. Component scores are \((\mu_k - \tilde{x})/v\): left \(-2.0\), right \(1.2\). Weighted:

A finite difference of \(\log p_\sigma\) confirms 0.94516; the energy \(-\log p_\sigma\) falls toward the heavier mode.

Step 4: one Langevin step with \(\epsilon = 0.1\) and noise draw \(z = -0.3\):

The noise scale \(\sqrt{2\epsilon} = 0.447\) is 4.5 times the drift coefficient, so one step moved away from the heavy mode. Drift wins only on average. At \(x_1\) the score is 0.9455, almost unchanged, which is why chains need hundreds of steps.

Step 5: Tweedie denoising. \(\mathbb{E}[x_0 \mid \tilde{x} = 0.5] = 0.5 + 1^2(0.9452) = 1.4452\). Check it as Bayes: within each component the posterior mean is \(\mu_k + (0.25/1.25)(\tilde{x} - \mu_k)\), giving \(-1.5\) and \(1.7\); mixing by responsibilities, \(0.0796(-1.5) + 0.9204(1.7) = 1.4452\). Identical. A noise-predicting network would output \(\hat\epsilon = -\sigma\nabla\log p_\sigma = -0.9452\).

Step 6: why annealing matters. Repeat Step 3 at other noise levels:

At \(\sigma = 0.1\), near \(x = 1.8\), the left responsibility is \(4 \times 10^{-13}\); swap the mixture weights and it becomes \(2 \times 10^{-12}\), leaving the score at 0.76923 either way. That is mode-weight blindness in numbers. At \(\sigma = 3\) the responsibilities (0.26, 0.74) approach the true weights on a nearly flat landscape, so chains visit modes in proportion to mass. High noise encodes weights; low noise encodes shape.

[IMAGE: Three stacked panels for sigma = 3, 1 and 0.1 showing the noised bimodal density and its score curve, with x = 0.5 marked and scores 0.051, 0.945 and 5.77 annotated. Caption: "Large sigma encodes mode weights, small sigma encodes mode shape."]

Where It Breaks

Scores are learned only near data

Denoising score matching supervises \(s_\theta\) only at noised training samples, which at small \(\sigma\) is a thin shell around the manifold. A trajectory pushed off it by discretisation error meets an untrained score. That is one reason EDM's \(\rho = 7\) schedule concentrates steps at low noise.

Score networks need not be gradients

A vector-valued network need not be conservative, so it may not be the gradient of any log density. Salimans and Ho tested whether explicit energies win on that basis and found "constrained score models, i.e. energy based models, can perform just as well as unconstrained models when using a comparable model structure" (Salimans & Ho, 2021, ICLR EBM Workshop). The disagreement moved to composition. Du et al. showed that combining diffusion models fails because of the sampler, and that an energy parameterisation enables Metropolis-corrected samplers that fix it (Du et al., 2023, ICML, arXiv:2302.11552). The mechanism: the sum of two models' scores at intermediate noise is not the score of the noised product distribution.

The ODE trusts the score completely

The probability-flow ODE matches the SDE's marginals only for an exact score. With an approximate one, errors accumulate along a deterministic path, while SDE noise partially forgets early mistakes. EDM's split result (churn hurt CIFAR-10, helped ImageNet-64) is this trade in practice; the authors grid-searched four churn parameters per model and warned that tuning stochastic samplers "may inadvertently end up influencing the design choices related to model architecture and training."

The objective memorises; networks generalise anyway

A perfect DSM minimiser on a finite training set is the score of Gaussians centred on training images, and sampling it reproduces them. Carlini et al. extracted over a thousand training examples from deployed diffusion models (Carlini et al., 2023, arXiv:2301.13188). Kadkhodaie et al. found generalisation emerging at "roughly \(10^5\) images" of CelebA, where networks trained on disjoint subsets learned nearly the same score (Kadkhodaie et al., 2024, ICLR, arXiv:2310.02557). Generalisation comes from the network's inductive bias; the objective is silent on it.

Alternative Designs

Flow matching is often sold as a replacement for score-based thinking. Lipman et al. note it "subsumes existing diffusion paths as specific instances" (Lipman et al., 2023, ICLR, arXiv:2210.02747); with Gaussian paths the velocity is an affine function of the denoiser. The real new freedom is non-diffusion paths such as straight-line interpolation, which Stable Diffusion 3 adopted as rectified flow (Esser et al., 2024, arXiv:2403.03206).

How It Is Used in Practice

Latent diffusion. Rombach et al. moved diffusion into a pretrained autoencoder's latent space because pixel-space training "often consumes hundreds of GPU days" (Rombach et al., 2022, CVPR, arXiv:2112.10752). The theory is untouched, but decoder quality now caps output quality and the latent's scale must be normalised or every preconditioning constant is wrong.

Inverse problems via Tweedie. A noise-conditional score is a prior. Diffusion Posterior Sampling computes the Tweedie estimate at each step and differentiates the measurement error through it, handling noisy nonlinear operators without retraining (Chung et al., 2023, ICLR, arXiv:2209.14687). The cost is an extra backward pass per step and step sizes that need per-task tuning.

Weather. GenCast is "implemented as a conditional diffusion model" trained with "a standard diffusion model denoising objective (Karras et al., 2022)." It produces a 15-day, 0.25° global ensemble in 8 minutes and reports more skill than ECMWF's ENS on 97.4% of 1,320 targets (Price et al., 2024, arXiv:2312.15796). An image recipe transferring to atmospheric fields supports the design-space view.

The sampler as a deployment knob. A trained denoiser fixes the score, not the integrator, so solvers swap without retraining.

[IMAGE: One trained denoiser in the centre with four sampler cards around it (ODE preview, SDE with churn, posterior sampling for inpainting, distilled one-step), each labelled with evaluations and relative latency. Caption: "One score, many integrators: the sampler is a deployment decision."]

Insights Worth Remembering

1. The normalising constant was made irrelevant, not beaten. Anything needing actual probabilities, such as likelihoods or Metropolis acceptance, pays for them again through the ODE's divergence integral or an explicit energy.

2. A denoiser is a well-conditioned score model. A denoiser's output stays image-scaled at every noise level while a raw score scales as \(1/\sigma\); EDM's preconditioning formalises that.

3. Noise is the representation, not a training trick. High noise carries mode weights, low noise carries shape. A single-level score model is not simpler diffusion; it cannot sample correctly.

4. Langevin, ancestral sampling and ODE solvers integrate one equation. The 2020 "diffusion versus score models" debate was about discretisation, and the reverse SDE and probability-flow ODE differ only by a factor of one half on the score term.

5. Energy parameterisation is a sampler feature, not a quality feature. At equal architecture energies match scores; they pay off when densities must be evaluated or composed.

6. The objective does not explain generalisation. A perfect DSM minimiser on finite data memorises; diffusion generalises because networks fail to fit the empirical score in an image-adapted way.

Open Questions

Why do networks learn a generalising score? Measured: networks trained on disjoint sets of about \(10^5\) CelebA images converge to nearly the same score (Kadkhodaie et al., 2024). Unknown: how to predict the memorisation threshold before training, especially for text-conditioned latent models.

How accurate must the score be? Shown: DDPM-style samplers converge in time polynomial in the problem parameters given an \(L^2\)-accurate score, without log-concavity assumptions (Chen et al., 2022, arXiv:2209.11215). Open: the bounds are worst-case, and nobody can measure a network's score error on real data, so theory does not yet say how many steps suffice.

When should sampling be stochastic? Evidence: churn hurt a strong CIFAR-10 model and helped an ImageNet-64 one. Speculation: it compensates for score error and should shrink as models improve.

Can explicit energies be trained at diffusion scale? Known: parity at matched architecture, and better compositional sampling. Unknown: whether the extra backward pass for \(\nabla_x E\) pays off in billion-parameter latent models.

Sources and Further Reading

1. Hyvärinen, A. (2005). "Estimation of Non-Normalized Statistical Models by Score Matching." JMLR, 6, 695-709. JMLR
2. Vincent, P. (2011). "A Connection Between Score Matching and Denoising Autoencoders." Neural Computation, 23(7), 1661-1674. MIT Press
3. Efron, B. (2011). "Tweedie's Formula and Selection Bias." JASA, 106(496), 1602-1614. doi
4. Anderson, B. D. O. (1982). "Reverse-time diffusion equation models." Stochastic Processes and their Applications, 12(3), 313-326. doi
5. Hinton, G. E. (2002). "Training Products of Experts by Minimizing Contrastive Divergence." Neural Computation, 14(8), 1771-1800. doi
6. Sohl-Dickstein, J., et al. (2015). "Deep Unsupervised Learning using Nonequilibrium Thermodynamics." ICML. arXiv:1503.03585
7. Song, Y., & Ermon, S. (2019). "Generative Modeling by Estimating Gradients of the Data Distribution." NeurIPS. arXiv:1907.05600
8. Song, Y., & Ermon, S. (2020). "Improved Techniques for Training Score-Based Generative Models." NeurIPS. arXiv:2006.09011
9. Ho, J., Jain, A., & Abbeel, P. (2020). "Denoising Diffusion Probabilistic Models." NeurIPS. arXiv:2006.11239
10. Song, Y., Sohl-Dickstein, J., Kingma, D. P., et al. (2021). "Score-Based Generative Modeling through Stochastic Differential Equations." ICLR. arXiv:2011.13456
11. Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). "Elucidating the Design Space of Diffusion-Based Generative Models." NeurIPS. arXiv:2206.00364
12. Du, Y., & Mordatch, I. (2019). "Implicit Generation and Modeling with Energy Based Models." NeurIPS. arXiv:1903.08689
13. Nijkamp, E., et al. (2019). "Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model." NeurIPS. arXiv:1904.09770; and (2020) "On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models." AAAI. arXiv:1903.12370
14. Salimans, T., & Ho, J. (2021). "Should EBMs Model the Energy or the Score?" ICLR Workshop on EBMs. ML Anthology. Counterpoint: Du, Y., et al. (2023). "Reduce, Reuse, Recycle." ICML. arXiv:2302.11552
15. Kadkhodaie, Z., & Simoncelli, E. P. (2021). "Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser." NeurIPS. arXiv:2007.13640; Kadkhodaie, Z., et al. (2024). "Generalization in diffusion models arises from geometry-adaptive harmonic representations." ICLR. arXiv:2310.02557
16. Fast sampling: Lu, C., et al. (2022). "DPM-Solver." NeurIPS. arXiv:2206.00927; Song, Y., et al. (2023). "Consistency Models." ICML. arXiv:2303.01469
17. Theory: Song, Y., et al. (2021), ScoreFlow, NeurIPS. arXiv:2101.09258; Chen, S., et al. (2022). arXiv:2209.11215; Song, Y., et al. (2019), Sliced Score Matching, UAI. arXiv:1905.07088
18. Applications and neighbours: Carlini, N., et al. (2023). arXiv:2301.13188; Rombach, R., et al. (2022), CVPR. arXiv:2112.10752; Chung, H., et al. (2023), ICLR. arXiv:2209.14687; Price, I., et al. (2024), GenCast. arXiv:2312.15796; Lipman, Y., et al. (2023), ICLR. arXiv:2210.02747; Esser, P., et al. (2024). arXiv:2403.03206

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.