Diffusion Models

A GAN trains a generator and a discriminator in adversarial equilibrium; the training is notoriously unstable and the samples often collapse to a few modes. Diffusion models replace the adversarial game with a much simpler problem: given an image with noise added, predict the noise. Iterate the denoising step and you have a generator. The objective is a clean regression, training is stable, the samples cover modes, and the resulting framework now underlies Stable Diffusion, DALL-E 3, Sora, Veo, Suno, and most of the generative-media stack.

The forward process

Pick a sequence of T timesteps and a noise schedule beta_1, ..., beta_T (typically linear or cosine, increasing). At each step you add a tiny bit of Gaussian noise to the image:

q(x_t | x_{t-1}) = N(x_t; sqrt(1 - beta_t) * x_{t-1}, beta_t * I)

A useful identity: you can jump straight from x_0 to x_t in closed form:

x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * eps,   eps ~ N(0, I)

where alpha_bar_t = prod_{s=1..t} (1 - beta_s). At t = T (typically 1000) the image is essentially pure noise. The forward process has no learnable parameters - it is a fixed corruption.

The reverse process as a learned vector field

The generator is the reverse of this chain. You want p_theta(x_{t-1} | x_t). Ho et al's 2020 DDPM showed that, given a small enough beta_t, the reverse step is also Gaussian, and the only thing you need to predict is the mean. Reparameterising, predicting the noise eps that was added is equivalent and works better in practice:

loss = E_{t, x_0, eps} || eps - eps_theta(x_t, t) ||^2

That is the entire training objective. Pick a random timestep, add the corresponding amount of noise to a clean image, ask the network to predict the noise. Sampling at inference is the reverse iteration: start from x_T ~ N(0, I), predict eps, subtract a scaled version, add a small fresh noise, repeat for 1000 steps. DDIM (Song et al, 2020) showed you can skip most steps and still get good samples - 20-50 steps is now standard, and distilled variants run in 1-4 steps.

DDPM vs the score-based view

There are two ways to look at the same algorithm:

• DDPM (Ho 2020). Discrete-time Markov chain; predict the noise.
• Score-based (Song & Ermon 2019, Song 2020). Continuous-time SDE; predict the score grad_x log p_t(x) of the noisy distribution.

The two formulations are equivalent up to reparameterisation. The score view connects diffusion to a body of older statistical theory (Langevin dynamics, score matching) and makes some manipulations (guidance, conditioning, accelerated sampling) cleaner. The DDPM view is operationally simpler. Most practitioners read the score-based paper for intuition and write the DDPM code.

U-Net vs DiT

The denoiser eps_theta(x_t, t) needs to be a network that takes an image and a timestep and returns an image-shaped noise prediction.

• U-Net. Encoder-decoder with skip connections, originally from medical image segmentation. The dominant choice in early diffusion (DDPM, Stable Diffusion ½, Imagen). Strong inductive bias for image-shaped outputs, good at low-medium scales.
• DiT (Diffusion Transformer). Peebles and Xie 2022 replaced the U-Net with a ViT-style transformer operating on patch tokens. Conditioning (timestep, class, text) enters via adaptive layer norm. DiTs scale better than U-Nets with compute and parameters - the FID-vs-FLOPs curve is steeper. Stable Diffusion 3, Flux, Sora, and Veo all use DiT-style backbones.

The pattern matches the CNN-to-ViT transition in classification: U-Nets had useful inductive bias; transformers scaled past them once compute and data justified the swap.

Classifier-free guidance

You want to condition generation on something - a class label, a text caption. The naive approach trains a separate classifier on noisy images and uses its gradients to steer sampling. It works but the classifier is hard to train.

Classifier-free guidance (Ho and Salimans 2022) sidesteps this:

1. During training, randomly drop the condition with probability ~10%. The model learns both eps_theta(x_t, t, c) (conditional) and eps_theta(x_t, t, null) (unconditional).
2. At sampling, compute both predictions and extrapolate:

eps_guided = eps_theta(x_t, t, null) + w * (eps_theta(x_t, t, c) - eps_theta(x_t, t, null))

w > 1 (typically 3-12) sharpens the conditional distribution - more prompt adherence, less diversity. w = 1 is the unmodified conditional. Every production text-to-image model uses CFG. The cost is two forward passes per sampling step instead of one; some distilled models bake CFG into the weights to avoid this.

Latent diffusion: Stable Diffusion's compute trick

Diffusion at pixel resolution is brutally expensive - a 1024x1024x3 image is 3M dimensions and you need to denoise it 50 times. Rombach et al (2022) noticed that most of those dimensions are imperceptible texture. Their fix:

1. Pretrain an autoencoder (VAE) that compresses 512x512 images to a 64x64x4 latent.
2. Train the diffusion model in this latent space instead of pixel space - 48x fewer dimensions.
3. At sampling, decode the final latent with the VAE.

The compute saving is 30-100x with negligible visual quality loss. This is what made Stable Diffusion runnable on a 12 GB consumer GPU and is the architectural foundation of every text-to-image model since. Most text-to-video models (Sora-class) do the same trick in 3D (spatial + temporal compression).

Flow Matching: the 2024+ alternative

Flow Matching (Lipman et al, 2022; Liu et al 2022 as Rectified Flow) reframes the problem. Instead of stepping through noise levels, learn a single vector field that transports samples from a simple prior (Gaussian) to the data distribution along straight (or near-straight) paths:

loss = E_{t, x_0, x_1} || v_theta(x_t, t) - (x_1 - x_0) ||^2

where x_t = (1 - t) * x_0 + t * x_1 and x_1 is real data, x_0 is noise. The integration path is by construction straighter than diffusion's curved SDE trajectory, which means fewer sampling steps for equivalent quality (4-16 vs 20-50). Stable Diffusion 3, Flux, and most 2024+ frontier image and video models are Flow Matching, not diffusion in the original sense. The two frameworks are deeply related (diffusion is a particular family of probability paths in the Flow Matching framework), and the code looks almost identical - the main difference is the training target and the sampler.

When it falls down

• Sampling speed. Even at 20 steps, a high-resolution diffusion model is slower than a single-pass GAN. Distillation (LCM, Hyper-SD) and few-step methods (Flow Matching, consistency models) close the gap to 1-4 steps but trade off some quality.
• Text rendering and structure. Pre-2024 models could not draw legible text or count fingers. The newer DiT-based models (Imagen 3, Flux, Stable Diffusion 3) largely fixed text; compositional binding (the red-cube-blue-sphere problem) remains rough.
• Controllability. Pure text conditioning is too low-bandwidth for many use cases. ControlNet, IP-Adapter, and LoRA bolt extra conditioning channels onto a frozen diffusion model; they are now standard in the production stack.

Further reading

• Denoising Diffusion Probabilistic Models - Ho, Jain, Abbeel 2020, the paper that started the modern diffusion era.
• High-Resolution Image Synthesis with Latent Diffusion Models - Rombach et al 2021, the Stable Diffusion paper.
• Scalable Diffusion Models with Transformers - Peebles and Xie 2022, the DiT paper.
• Classifier-Free Diffusion Guidance - Ho and Salimans, the CFG paper.
• Flow Matching for Generative Modeling - Lipman et al 2022, the framework that displaced classical diffusion in many 2024 models.
• What are Diffusion Models? - Lilian Weng's reference-quality walkthrough of the maths.
• The Annotated Diffusion Model - Hugging Face's code-first DDPM tutorial.