Classifier-Free Guidance
The extrapolation trick that made text-to-image work, why it is not sampling from any distribution the model learned, and the fidelity-diversity tradeoff it exposes as a single tunable number.
A conditional diffusion model trained on text-image pairs, sampled honestly from \(p(x \mid c)\), produces images loosely related to the prompt and disappointingly bland. Every production text-to-image system therefore does something else: it samples from a distribution the model was never trained on, obtained by extrapolating away from the unconditional prediction. That single hack separates a research curiosity from a product.
The mechanism
Train one network to handle both conditional and unconditional prediction by randomly dropping the conditioning during training, typically around 10 percent of the time, replacing it with a learned null embedding. At sampling time, evaluate both and extrapolate:
With \(w = 1\) this is ordinary conditional sampling. With \(w > 1\) the update moves further along the direction conditioning added, past the conditional estimate itself. Values between 5 and 8 are typical for text-to-image; the method comes from Ho and Salimans, 2022, Classifier-Free Diffusion Guidance, arXiv:2207.12598.
The name is historical. The predecessor, classifier guidance, required training a separate noise-aware classifier and using its gradient to steer sampling, which meant maintaining a second model robust at every noise level. Classifier-free guidance removes that model, and the score-function identity shows why the two are equivalent: the difference between conditional and unconditional score estimates is exactly the gradient of the implicit classifier \(\log p(c \mid x_t)\).
What it costs
Two forward passes per step. Guidance doubles inference compute, which for a large model is the dominant cost of image generation. Batching the conditional and unconditional inputs together recovers some of it through better GPU utilisation, but the FLOPs are real, and eliminating them is the point of distilled-guidance methods that fold guided behaviour into a single conditional pass.
Diversity. Guidance sharpens the conditional distribution, concentrating mass on the modes that most strongly satisfy the prompt. At \(w=2\) you get variety; at \(w=15\) every sample of "a photograph of a dog" converges toward the same well-lit, centred, high-contrast dog. The metrics show this directly: raising \(w\) improves prompt-alignment scores such as CLIP score while worsening FID past a moderate value, because FID penalises lost coverage. Reporting a single FID for a guided model without stating \(w\) is close to meaningless; the honest presentation is a curve traced by sweeping \(w\).
Saturation and artefacts. High guidance pushes predicted \(x_0\) outside the valid pixel range. The visible result is blown-out highlights, oversaturated colour, and a characteristic plastic sheen. Dynamic thresholding, introduced with Imagen, rescales the prediction based on a high percentile of its absolute values each step and lets much higher guidance scales be used without the artefacts.
Refinements worth knowing
Guidance scale need not be constant across the trajectory. It matters most in the middle of the chain, where global composition is decided, and least at the very start and end. Ramping \(w\) up and then down recovers detail and diversity that a constant scale destroys.
Negative prompting is guidance with the null embedding replaced by an embedding of unwanted content, so the extrapolation moves away from a specific region rather than away from the model's unconditioned average. Same arithmetic, different origin, and it costs nothing extra.
When it breaks
The guided distribution is not one the model represents. There is no \(p\) for which the guided sampler is a correct sampler; the extrapolated score does not integrate to a normalised density. Guarantees derived for the diffusion ODE, including exact likelihoods and clean invertibility, do not survive guidance, which is why inversion-based editing degrades at high guidance.
Interaction with the solver. As with higher-order samplers, strong guidance stiffens the trajectory and destabilises solvers that assume smoothness. A guidance scale that works at 50 steps can produce visible artefacts at 15.
It masks a weak conditional model. Needing \(w=12\) for prompt adherence is evidence that the conditional signal is weak, often because the text encoder is undertrained or the caption data is poor. Turning up guidance treats the symptom and costs diversity; improving captions treats the cause. Recaptioning training data with a vision-language model, as several systems have reported, reduces the guidance scale required.
It does not transfer across model families. Flow-matching and rectified-flow models use guidance too, but the numerically equivalent scale differs, so carrying a favourite value across architectures produces either weak conditioning or heavy artefacts.
14 flashcards for this concept
Click a card to reveal the answer.