Energy-Based & Score Models
Unnormalised densities, score matching, Langevin dynamics, and the SDE view that unifies the generative families.
11concepts
99flashcards
86minutes of reading
- 01 Hopfield Networks and Attractor Memory How a quadratic energy function turns stored patterns into attractors that can be recalled from corrupted fragments, why the usable capacity is only about 0.138 times the number of units, and what the network returns once you exceed it.
- 02 Annealed Importance Sampling and Estimating log Z How a chain of bridging distributions turns an intractable partition function into an unbiased ratio estimate, why that makes reported log-likelihoods too high rather than too low, and how reverse annealing bounds the error from the other side.
- 03 Boltzmann Machines and Hidden-Unit Energies How adding temperature and hidden units turns an attractor network into a generative distribution, why the restricted architecture makes sampling cheap, and the exact mapping that shows a Hopfield network is an RBM with Gaussian hidden units.
- 04 Contrastive Divergence and the Negative Phase How energy-based training approximates an intractable expectation with a short MCMC chain, what bias that introduces, and why the same positive-negative structure appears in contrastive learning and reward modelling.
- 05 Dense Associative Memory and the Sharpness of Separation How replacing the quadratic energy with a sharper function lifts storage capacity from 0.138N to a polynomial or exponential function of N, why the mechanism is amplification of the largest similarity, and what that sharpness costs.
- 06 Energy-Based Models and the Partition Function The most flexible way to specify a probability distribution, why the normalising constant makes it untrainable by direct likelihood, and the three escape routes that define the rest of the field.
- 07 Langevin Dynamics for Sampling How a noisy gradient ascent on log density becomes a valid sampler, why the noise term is what separates sampling from optimisation, and the mixing failure that makes it impractical alone.
- 08 Modern Hopfield Networks as Attention How the log-sum-exp energy over continuous states gives an update rule identical to transformer attention, what the three kinds of fixed point mean for a real network, and where the equivalence stops holding.
- 09 Noise-Contrastive Estimation How training a classifier to tell data from known noise recovers an unnormalised density and its normalising constant, why the choice of noise decides whether that works, and how word2vec's negative sampling and InfoNCE descend from it.
- 10 Score Matching and Its Denoising Form How targeting the gradient of log density eliminates the partition function, why the naive form requires an intractable Hessian trace, and how adding noise makes the objective a simple regression.
- 11 The SDE View of Generative Models The continuous-time framework in which diffusion, score matching and denoising are the same object, why every SDE has a deterministic twin with identical marginals, and what the unification actually buys.