Variational & Flow Models
The ELBO, reparameterisation, normalising flows, flow matching and rectified transport.
11concepts
101flashcards
82minutes of reading
- 01 Amortised Inference and the Amortisation Gap Why a VAE's encoder is a shortcut rather than an inference algorithm, how to measure the two distinct ways its posterior falls short of the truth, and what to do when the shortcut is the thing holding your model back.
- 02 The Reparameterisation Trick Why you cannot backpropagate through a sampling operation, how moving the randomness to an input fixes it, and the variance argument that explains why this beats the score-function estimator.
- 03 Autoregressive Flows: MAF and IAF Why the same masked autoregressive transformation is fast in one direction and sequential in the other, how that single asymmetry splits the flow literature into density estimators and samplers, and how distillation gets both.
- 04 Continuous Normalising Flows and Neural ODEs How defining a flow as the solution of an ODE replaces the Jacobian determinant with a trace, why Hutchinson's estimator makes that trace linear in dimension, what the adjoint method buys and costs, and why simulation in the training loop kept these models slow.
- 05 Dequantisation and Likelihoods on Discrete Data Why fitting a continuous density to integer-valued data diverges, how adding noise converts the problem into a valid lower bound on the discrete likelihood, and how much that bound costs when the noise is chosen badly.
- 06 Flow Matching and Conditional Vector Fields How to train a continuous-time generative model by regressing a velocity field without ever simulating an ODE, and why conditioning on a single data point makes an intractable target tractable.
- 07 Importance-Weighted Bounds and the SNR Problem How averaging importance weights inside the logarithm gives a strictly tighter bound on the evidence, and the result that this tighter bound degrades the encoder's gradient signal-to-noise ratio as the sample count grows.
- 08 Normalising Flows and the Cost of Invertibility How change of variables turns a simple density into a complex one with an exact likelihood, why the Jacobian determinant constrains every architectural choice, and what that constraint costs relative to VAEs and diffusion.
- 09 Posterior Collapse and the KL Term The failure where a VAE's latent code carries no information, why a powerful decoder makes it the optimal solution rather than an accident, and the fixes that each buy something different.
- 10 Rectified Flow and Trajectory Straightening Why the generative ODE learned from independent noise-data pairs is curved, how reflow iteratively straightens it, and what each round of straightening costs in fidelity.
- 11 The Rate-Distortion View of the ELBO Why the ELBO is a sum of two quantities that trade off against each other, how an entire family of models can share one ELBO while differing completely in what their latents encode, and what that means for choosing beta.