Denoising Parameterisations: Epsilon, x-zero and v
Three algebraically equivalent things a diffusion network can predict, why they train to completely different models, and how the choice interacts with the noise schedule and with distillation.
Given a noisy sample \(x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon\), knowing any one of \(x_0\), \(\epsilon\) or \(x_t\) determines the other two. So it should not matter which the network predicts. It matters enormously, because the three targets are conditioned differently at different noise levels, and the training loss weights errors differently in each case.
The three targets
Epsilon prediction trains \(\epsilon_\theta(x_t, t)\) against the noise that was added, with the objective \(\mathbb{E}\lVert \epsilon - \epsilon_\theta(x_t,t)\rVert^2\). This is the DDPM formulation and it dominated early work. Its virtue is that the target has unit variance at every \(t\), so the loss scale is stable across the whole chain and no per-timestep normalisation is needed.
x-zero prediction trains the network to output the clean sample directly. Recovering \(\epsilon\) from it requires dividing by \(\sqrt{1-\bar{\alpha}_t}\), which at low \(t\) is close to zero, so a small error in \(\hat{x}_0\) becomes an enormous error in the implied noise. At high \(t\), conversely, predicting \(x_0\) from near-pure noise is close to hopeless and the network's best answer is the dataset mean.
v-prediction interpolates between them: \(v_t = \sqrt{\bar{\alpha}_t}\,\epsilon - \sqrt{1-\bar{\alpha}_t}\,x_0\). It equals \(\epsilon\) at \(t=T\) and \(-x_0\) at \(t=0\), rotating smoothly in between. Introduced for progressive distillation (Salimans and Ho, 2022, arXiv:2202.00512), it is now the default in most high-resolution and video systems.
Why the choice changes the model
Rewrite each objective as a weighted version of the same underlying quantity and the difference becomes a per-timestep loss weight. Predicting \(\epsilon\) and predicting \(x_0\) differ by a factor of \(\text{SNR}(t)\) in how heavily each timestep contributes to the gradient. Epsilon prediction up-weights the low-SNR end, where the model learns global structure; x-zero prediction up-weights the high-SNR end, where it learns texture. Neither is universally right, and the practical schedules in use are largely attempts to correct one parameterisation's implicit weighting back toward something balanced.
The failure of x-zero prediction at high noise has a concrete signature. The optimal prediction of \(x_0\) given nearly pure noise is \(\mathbb{E}[x_0]\), a grey blur, and a network trained on that objective spends capacity learning to output blurs. Epsilon prediction has the mirror-image problem as \(t \to 0\): the noise is almost all that is left in \(x_t\), so predicting it is nearly an identity map and the gradient carries little information about the data distribution.
v-prediction avoids both degeneracies because its target has unit variance at every \(t\) and remains a non-trivial function of the data at both ends of the chain. This is why it survives aggressive step-count reduction: at two or four sampling steps, epsilon prediction's near-identity behaviour at low noise produces visible artefacts, and v does not.
Interaction with zero terminal SNR
Parameterisation and schedule are not independent choices. Fix the schedule so \(\bar{\alpha}_T = 0\) exactly and epsilon prediction becomes degenerate at the final step: \(x_T\) is \(\epsilon\), so the network is asked to output its own input and the reverse step carries no information. v-prediction has no such problem, since at \(\bar{\alpha}_T = 0\) it reduces to \(\epsilon\) smoothly while the implied estimate of \(x_0\) remains meaningful. Any implementation that adopts zero terminal SNR while keeping epsilon prediction has to special-case the last step, which is a reliable sign the two choices should have been made together.
When it breaks
Mixed conventions in a codebase. The same checkpoint interpreted under the wrong parameterisation produces output that is recognisably image-like but consistently wrong in contrast and saturation, rather than obviously broken. This is a common bug when porting weights between repositories, and it is hard to spot because the samples are not noise.
Loss weighting silently reintroduces the problem. Many implementations train with epsilon prediction and then apply a min-SNR or truncated-SNR weighting to the loss. That weighting is doing the job the parameterisation should have done, and stacking it on top of v-prediction, which already has balanced weighting, can over-correct and starve the low-noise end.
Distillation is where the difference becomes visible. At 50 sampling steps the three parameterisations produce similar quality and the argument looks academic. Progressive distillation down to four or two steps forces each step to cover a wide SNR range, and only v-prediction stays well conditioned across it. Choosing epsilon prediction is a decision to make later distillation harder, taken long before anyone plans to distil.
Conditional generation shifts the balance. With strong conditioning, the high-noise end becomes easier because the text prompt already specifies global layout. That changes where capacity is best spent and weakens the usual argument for epsilon prediction.
14 flashcards for this concept
Click a card to reveal the answer.