Energy-Based Models and the Partition Function
The most flexible way to specify a probability distribution, why the normalising constant makes it untrainable by direct likelihood, and the three escape routes that define the rest of the field.
Every other generative model in this library is built around a constraint. A VAE needs a tractable posterior approximation. A flow needs invertibility. An autoregressive model needs an ordering. An energy-based model has none: take any function \(E_\theta(x)\) that maps an input to a scalar, and it defines a distribution. That freedom is the appeal, and the price is a single integral that cannot be computed.
The definition
Low energy means high probability. \(E_\theta\) can be any neural network with a scalar output, with no architectural constraint whatsoever, which is a stronger statement than it first appears: a ResNet, a transformer, a physics simulator, or a hand-written scoring function all qualify.
\(Z(\theta)\), the partition function, is an integral over the entire input space. For a 32x32 colour image that is an integral over \(\mathbb{R}^{3072}\). It has no closed form for any interesting \(E_\theta\), and it depends on \(\theta\), so it cannot be treated as a constant during training.
Why the gradient is still tractable in principle
Take the gradient of the log-likelihood and the intractable term does not disappear, but it turns into something with a sampling interpretation:
The first term is the positive phase: push down the energy of the observed data point. The second is the negative phase: push up the energy of samples drawn from the model's own current distribution. Together they carve a well around the data and flatten everything else.
This is a genuinely useful decomposition, and it relocates the problem rather than solving it. The negative phase requires sampling from \(p_\theta\), which requires MCMC, which requires many evaluations of \(E_\theta\) per training step and may not have mixed by the time you stop.
The three escape routes
Approximate the negative phase. Contrastive divergence runs a short MCMC chain from the data rather than from equilibrium, accepting bias in exchange for tractability. Persistent contrastive divergence maintains chains across parameter updates so they have longer to mix. Both work and both are delicate.
Change the objective so \(Z\) cancels. Score matching targets \(\nabla_x \log p_\theta(x)\), and since \(Z\) does not depend on \(x\), that gradient is exactly \(-\nabla_x E_\theta(x)\) with the partition function gone. This is the route that led to diffusion models.
Give up on normalisation. Many applications need only relative comparisons: ranking candidates, scoring plausibility, or providing a term in a larger objective. Unnormalised energies are sufficient for all of them, and this is where energy-based reasoning quietly shows up inside models that are not called energy-based, including the reward models used in RLHF.
When it breaks
MCMC in high dimensions does not mix. The chains used in practice are far too short to visit the modes of a high-dimensional distribution, so the negative phase samples come from wherever the chain happens to be rather than from \(p_\theta\). Training is then optimising a biased gradient, and the bias is neither small nor well characterised.
Training is unstable without explicit regularisation. Nothing bounds the energy function, so training can push energies to extreme values in regions the chains never visit, and the model diverges. Practical implementations add an L2 penalty on energy magnitudes, clip gradients aggressively, and inject noise into MCMC steps. These are load-bearing, not tuning details.
Likelihood is not comparable. Because \(Z\) is unknown, an energy-based model cannot report a log-likelihood without estimating it, typically by annealed importance sampling, which is expensive and itself has variance that grows with dimension. Reported likelihoods for these models should be read as estimates with wide error bars.
The flexibility is real and mostly used elsewhere. The clearest wins are compositional: because energies add, combining two constraints is \(E_1 + E_2\), which gives a principled way to compose models that no other family offers as cleanly. Applications in protein design, physical systems with known energy terms, and structured prediction exploit this. Pure image generation, the benchmark the field measured itself against for years, is where energy-based models are least competitive.
12 flashcards for this concept
Click a card to reveal the answer.