Concept library
1134 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
13
Generative Modelling Beyond Transformers
Diffusion, flows, adversarial games and state space models, and the theory that connects them.
5tracks
47concepts
452cards
5.9hreading
Diffusion Models Forward noising, denoising objectives, samplers, classifier-free guidance and latent diffusion. 7 concepts · 82 cards
- 01 Classifier-Free Guidance The extrapolation trick that made text-to-image work, why it is not sampling from any distribution the model learned, and the fidelity-diversity tradeoff it exposes as a single tunable number.
- 02 Forward Diffusion and Noise Schedules The fixed corruption process that makes diffusion training possible, why the closed form in alpha-bar removes the need to simulate it, and how the schedule silently decides which frequencies the model learns.
- 03 Latent Diffusion and the Autoencoder Bottleneck Why running diffusion in a compressed latent space cuts cost by more than an order of magnitude, what the autoencoder throws away permanently, and the failure modes that belong to the VAE rather than the diffusion model.
- 04 Consistency Models and Few-Step Sampling How consistency models learn a direct map from any point on a probability-flow trajectory to its origin, the difference between distilling that map from a diffusion teacher and training it from scratch, and how latent and adversarial variants reach one to four steps.
- 05 Denoising Parameterisations: Epsilon, x-zero and v Three algebraically equivalent things a diffusion network can predict, why they train to completely different models, and how the choice interacts with the noise schedule and with distillation.
- 06 Diffusion Transformers (DiT) Why replacing the diffusion U-Net with a plain vision transformer over latent patches made image quality a predictable function of forward-pass compute, and how adaLN-Zero conditioning and the MM-DiT joint-attention block made it work.
- 07 Samplers: DDPM, DDIM and Higher-Order Solvers Why sampling is numerical integration of an ODE or SDE rather than a fixed algorithm, how DDIM makes the process deterministic and skippable, and what higher-order solvers buy at the cost of stability.
Variational & Flow Models The ELBO, reparameterisation, normalising flows, flow matching and rectified transport. 11 concepts · 101 cards
- 01 Amortised Inference and the Amortisation Gap Why a VAE's encoder is a shortcut rather than an inference algorithm, how to measure the two distinct ways its posterior falls short of the truth, and what to do when the shortcut is the thing holding your model back.
- 02 The Reparameterisation Trick Why you cannot backpropagate through a sampling operation, how moving the randomness to an input fixes it, and the variance argument that explains why this beats the score-function estimator.
- 03 Autoregressive Flows: MAF and IAF Why the same masked autoregressive transformation is fast in one direction and sequential in the other, how that single asymmetry splits the flow literature into density estimators and samplers, and how distillation gets both.
- 04 Continuous Normalising Flows and Neural ODEs How defining a flow as the solution of an ODE replaces the Jacobian determinant with a trace, why Hutchinson's estimator makes that trace linear in dimension, what the adjoint method buys and costs, and why simulation in the training loop kept these models slow.
- 05 Dequantisation and Likelihoods on Discrete Data Why fitting a continuous density to integer-valued data diverges, how adding noise converts the problem into a valid lower bound on the discrete likelihood, and how much that bound costs when the noise is chosen badly.
- 06 Flow Matching and Conditional Vector Fields How to train a continuous-time generative model by regressing a velocity field without ever simulating an ODE, and why conditioning on a single data point makes an intractable target tractable.
- 07 Importance-Weighted Bounds and the SNR Problem How averaging importance weights inside the logarithm gives a strictly tighter bound on the evidence, and the result that this tighter bound degrades the encoder's gradient signal-to-noise ratio as the sample count grows.
- 08 Normalising Flows and the Cost of Invertibility How change of variables turns a simple density into a complex one with an exact likelihood, why the Jacobian determinant constrains every architectural choice, and what that constraint costs relative to VAEs and diffusion.
- 09 Posterior Collapse and the KL Term The failure where a VAE's latent code carries no information, why a powerful decoder makes it the optimal solution rather than an accident, and the fixes that each buy something different.
- 10 Rectified Flow and Trajectory Straightening Why the generative ODE learned from independent noise-data pairs is curved, how reflow iteratively straightens it, and what each round of straightening costs in fidelity.
- 11 The Rate-Distortion View of the ELBO Why the ELBO is a sum of two quantities that trade off against each other, how an entire family of models can share one ELBO while differing completely in what their latents encode, and what that means for choosing beta.
Adversarial Generative Models The minimax game, mode collapse, Wasserstein critics, and what FID does and does not measure. 7 concepts · 76 cards
- 01 Adversarial Losses as a Component Why GANs largely lost as standalone generative models but their loss survived inside autoencoders, vocoders and super-resolution, and what a patch discriminator adds that L2 cannot.
- 02 Conditional GANs and Image-to-Image Translation How conditioning information enters a GAN's generator and discriminator, why pix2pix needs paired data while CycleGAN substitutes cycle consistency, what that substitute silently permits, and why the projection discriminator beat concatenation.
- 03 The GAN Minimax Game What the discriminator is actually estimating, why the theoretically clean generator loss cannot be used in practice, and what it means that training seeks an equilibrium rather than a minimum.
- 04 What FID Measures and What It Misses The Gaussian assumption inside Frechet Inception Distance, the sample-size bias that makes numbers incomparable across papers, and why precision and recall metrics exist.
- 05 Mode Collapse and Training Instability Why a GAN generator has no incentive to cover the data distribution, the difference between full and partial collapse, and what each of the standard mitigations actually changes.
- 06 StyleGAN and Style-Based Generation How StyleGAN moved the latent code out of the generator's input and into per-layer modulation through a learned mapping network, and how StyleGAN2 and StyleGAN3 traced visible artefacts back to normalisation, progressive growing and aliasing.
- 07 Wasserstein Critics and the Lipschitz Constraint Why earth mover distance still has a gradient when JS divergence does not, how the Kantorovich duality turns it into a trainable critic, and why enforcing the Lipschitz bound is where every practical difficulty lives.
State Space Models S4, Mamba, selective scan, the recurrence-convolution duality, and where linear-time sequence models pay off. 11 concepts · 94 cards
- 01 Hybrid Attention-SSM Architectures Why the strongest linear-time models are not pure state space models, what a small number of full attention layers restores, and how to reason about the mixing ratio.
- 02 The Fixed State Budget at Inference Why a state space model decodes in constant memory per sequence, how to do the arithmetic against a KV cache, and the operational consequences that constant memory buys and costs.
- 03 Discretising Continuous State Spaces How a continuous-time state space layer becomes a recurrence you can run on tokens, why zero-order hold and the bilinear transform disagree at large step sizes, and why the step size ends up acting as a learned timescale and a gate.
- 04 Distilling Transformers into Linear-Time Models How a pretrained transformer's attention projections are reused to initialise state space layers, why the conversion is staged rather than end to end, and what the resulting hybrids do and do not recover.
- 05 HiPPO and Structured State Initialisation Why a randomly initialised state space layer fails on long sequences, what optimal polynomial projection of history gives you instead, and how the structure that makes it work also makes it computable.
- 06 Long Convolution Models and Implicit Filters How H3 and Hyena reached subquadratic sequence mixing through FFT convolutions with implicitly parameterised filters, what data-controlled gating contributed, and why selectivity displaced the whole family.
- 07 Parallel Scan and Hardware-Aware SSM Kernels How an associative scan recovers training parallelism after selectivity destroys the convolution, and why the arithmetic-intensity argument means the kernel is the architecture.
- 08 Selectivity and Input-Dependent Parameters What linear time-invariant models fundamentally cannot do, how making the state space parameters functions of the input fixes it, and the computational bill that change immediately creates.
- 09 State Capacity and Associative Recall The information-theoretic reason a fixed-size recurrent state cannot copy arbitrary context, what MQAR measures that perplexity hides, and why the gap shows up on exactly the tasks production systems care about.
- 10 Structured State Space Duality The equivalence between selective state space models and a masked form of linear attention, why writing the recurrence as a semiseparable matrix recovers tensor-core throughput, and what the chunked algorithm actually computes.
- 11 The Recurrence-Convolution Duality Why a linear state space layer can be run as a parallel convolution during training and as a constant-memory recurrence at inference, and what the word "linear" is buying.
Energy-Based & Score Models Unnormalised densities, score matching, Langevin dynamics, and the SDE view that unifies the generative families. 11 concepts · 99 cards
- 01 Hopfield Networks and Attractor Memory How a quadratic energy function turns stored patterns into attractors that can be recalled from corrupted fragments, why the usable capacity is only about 0.138 times the number of units, and what the network returns once you exceed it.
- 02 Annealed Importance Sampling and Estimating log Z How a chain of bridging distributions turns an intractable partition function into an unbiased ratio estimate, why that makes reported log-likelihoods too high rather than too low, and how reverse annealing bounds the error from the other side.
- 03 Boltzmann Machines and Hidden-Unit Energies How adding temperature and hidden units turns an attractor network into a generative distribution, why the restricted architecture makes sampling cheap, and the exact mapping that shows a Hopfield network is an RBM with Gaussian hidden units.
- 04 Contrastive Divergence and the Negative Phase How energy-based training approximates an intractable expectation with a short MCMC chain, what bias that introduces, and why the same positive-negative structure appears in contrastive learning and reward modelling.
- 05 Dense Associative Memory and the Sharpness of Separation How replacing the quadratic energy with a sharper function lifts storage capacity from 0.138N to a polynomial or exponential function of N, why the mechanism is amplification of the largest similarity, and what that sharpness costs.
- 06 Energy-Based Models and the Partition Function The most flexible way to specify a probability distribution, why the normalising constant makes it untrainable by direct likelihood, and the three escape routes that define the rest of the field.
- 07 Langevin Dynamics for Sampling How a noisy gradient ascent on log density becomes a valid sampler, why the noise term is what separates sampling from optimisation, and the mixing failure that makes it impractical alone.
- 08 Modern Hopfield Networks as Attention How the log-sum-exp energy over continuous states gives an update rule identical to transformer attention, what the three kinds of fixed point mean for a real network, and where the equivalence stops holding.
- 09 Noise-Contrastive Estimation How training a classifier to tell data from known noise recovers an unnormalised density and its normalising constant, why the choice of noise decides whether that works, and how word2vec's negative sampling and InfoNCE descend from it.
- 10 Score Matching and Its Denoising Form How targeting the gradient of log density eliminates the partition function, why the naive form requires an intractable Hessian trace, and how adding noise makes the objective a simple regression.
- 11 The SDE View of Generative Models The continuous-time framework in which diffusion, score matching and denoising are the same object, why every SDE has a deterministic twin with identical marginals, and what the unification actually buys.