Consistency Models and Few-Step Sampling

A good ODE sampler needs somewhere between 10 and 50 network evaluations to turn noise into an image, as the samplers concept explains. Each evaluation of a billion-parameter backbone costs real latency. Consistency models attack the count directly: instead of learning a velocity or noise field and integrating it, they learn the answer the integration would have produced. On CIFAR-10 the original method produced FID 3.55 from a single forward pass (Song, Dhariwal, Chen & Sutskever, 2023, Consistency Models, ICML, arXiv:2303.01469).

The self-consistency property

A diffusion model defines a probability-flow ODE whose trajectories \(\{x_t\}_{t \in [\epsilon, T]}\) run deterministically from noise at \(t = T\) to data at \(t = \epsilon\). Every point on one trajectory shares the same endpoint. A consistency function \(f\) maps each point to it:

with the boundary condition \(f(x_\epsilon, \epsilon) = x_\epsilon\). The boundary condition is built into the architecture rather than learned:

where \(F_\theta\) is any free-form network, typically a diffusion U-Net or transformer. Sampling draws \(\hat{x}_T \sim \mathcal{N}(0, T^2 I)\) and returns \(f_\theta(\hat{x}_T, T)\). For better quality, a multistep sampler re-noises the output to an intermediate time and applies \(f_\theta\) again, so the compute dial survives: one step for speed, two or three for fidelity.

Distillation versus training

Nothing tells the network where trajectories go unless something supplies adjacent pairs of points on the same trajectory. The two methods differ in where those pairs come from.

Consistency distillation (CD) uses a pretrained diffusion model \(\phi\). Sample data \(x\), noise it to \(x_{t_{n+1}}\), take one ODE-solver step with the teacher to get \(\hat{x}^\phi_{t_n}\), and minimise

where \(d\) is a distance (squared \(\ell_2\), \(\ell_1\) or LPIPS), \(\lambda\) a weighting, and \(\theta^-\) an exponential moving average of \(\theta\) held under stop-gradient as a target network. The best CIFAR-10 configuration used LPIPS, Heun's solver and \(N = 18\) discretisation steps. The paper proves that if this loss reaches zero, the model's error against the true consistency function is \(O((\Delta t)^p)\) for a solver of order \(p\).

Consistency training (CT) drops the teacher. It uses the same data point and the same Gaussian draw \(z\) at two noise levels, \(x + t_n z\) and \(x + t_{n+1} z\), a substitute for the teacher step that the paper justifies as the discretisation becomes fine. Originally CT lagged far behind: one-step FID 8.70 on CIFAR-10 against CD's 3.55.

That gap was not permanent. Song & Dhariwal, 2023, Improved Techniques for Training Consistency Models, arXiv:2310.14189 removed the EMA from the teacher branch after identifying it as a flaw in CT's theory, replaced LPIPS with a pseudo-Huber loss, used a lognormal noise schedule and doubled the discretisation steps on a fixed cadence. One-step FID fell to 2.51 on CIFAR-10 and 3.25 on ImageNet 64×64, beating distillation. The authors changed their own minds within the year: the EMA target was first presented as a stabiliser, then argued to be a source of bias in the training theory. The same paper argued LPIPS can inflate FID, because the LPIPS network and the Inception network inside FID are both trained on ImageNet.

Latent and adversarial variants

Latent consistency models apply CD inside Stable Diffusion's latent space (Luo et al., 2023, Latent Consistency Models, arXiv:2310.04378). Two changes matter. Classifier-free guidance, which normally doubles cost per step, is folded into the target by distilling an augmented ODE whose field already includes the guidance scale \(\omega\), and the model takes \(\omega\) as an input. And the skipping-step trick enforces consistency between \(t_{n+k}\) and \(t_n\) with \(k = 20\) rather than between adjacent steps, shortening the schedule. A 2 to 4 step 768×768 model cost about 32 A100 GPU-hours and 4,000 training steps.

Adversarial diffusion distillation takes a different route (Sauer et al., 2023, Adversarial Diffusion Distillation, arXiv:2311.17042). A student initialised from the diffusion U-Net denoises from four chosen timesteps, the last at pure noise. Its loss is \(\mathcal{L}_{\text{adv}} + \lambda \mathcal{L}_{\text{distill}}\) with \(\lambda = 2.5\): a hinge GAN loss from lightweight heads on a frozen DINOv2 feature network, plus a score-distillation term that re-noises the student's output and matches the teacher's denoised prediction. By the authors' evaluation, the single-step model beats LCM-XL and four steps reach the performance of SDXL itself. The discriminator is there because distance-based targets alone reward averages, and averages of plausible images are blurry.

When it breaks

Distillation caps quality at the teacher. CD cannot exceed the model it copies, and any bias in the teacher's ODE is inherited.

Consistency training is sensitive to its schedule. The original paper found that a small \(N\) converges faster to worse samples, while a large \(N\) converges slower to better ones; the improved recipe is a hand-built curriculum around exactly that trade.

Extra steps buy little. Re-noising and re-applying the model is not integration, so it lacks a solver's convergence order; in the improved recipe a second step moved CIFAR-10 FID only from 2.51 to 2.24. Anyone expecting the long quality dial of a diffusion sampler will find a short one.

Adversarial terms can bring GAN failure modes back. A discriminator rewards samples that look real, not samples that cover the distribution, and ADD's own ablations show quality depending on which frozen feature network the discriminator uses, with DINOv2 best. The sharpness is bought with a component whose behaviour the diffusion objective no longer controls.