IPO and the Overfitting Fix

DPO reduced RLHF to a binary classification problem, shipped in a weekend, and outperformed PPO on several benchmarks. The community celebrated. Then Azar et al. (2023) pointed out that DPO's core loss has a theoretical failure mode: given infinite gradient steps, it will drive the policy to assign zero probability to every rejected completion, regardless of what the preference data actually says. This is not a hyperparameter problem. It is a structural consequence of the sigmoid transform.

IPO (Identity Preference Optimisation) is the fix. It replaces one line in the DPO loss, costs nothing extra at inference, and comes with a formal performance guarantee that DPO lacks. Understanding why this replacement is necessary requires tracing back through a chain of approximations most practitioners skip.

The two approximations buried inside DPO

DPO's elegant derivation hides two load-bearing assumptions.

Assumption 1: Pairwise preferences decompose into pointwise rewards. DPO uses the Bradley-Terry model, which says the probability that response \(y^+\) is preferred over \(y^-\) given prompt \(x\) is:

This is sensible for stochastic annotators. But it implies there exists a scalar reward \(r(x, y)\) for each response independently. Real human preferences are often context-dependent and non-transitive; the Bradley-Terry model smooths that complexity away.

Assumption 2: The reward estimated on the training distribution generalises to the policy distribution. DPO eliminates the reward model by re-expressing \(r\) in terms of the optimal policy and reference model. The trick only holds if the policy and reference are close enough that the reward learned from labelled pairs transfers. On small datasets, or after many gradient steps, the policy drifts far enough from the reference that the implicit reward is no longer a reliable signal.

These assumptions work reasonably well in practice. The problem is what the DPO loss does near convergence.

Why DPO can overfit to deterministic preferences

The DPO objective is:

Define the implicit reward margin as:

The loss decreases monotonically as \(h_\theta \to +\infty\). There is no plateau. The global minimum of the loss is achieved only when \(\pi_\theta(y^- \mid x) \to 0\), which collapses the rejected completions entirely.

This matters because the training data is finite. Preference pairs are labelled examples, not samples from an infinite oracle. When you have a fixed dataset and unconstrained gradient steps, DPO will memorise the training preferences by suppressing rejected probabilities to near-zero, a pathological regime where the model has effectively assigned infinite reward to chosen completions relative to rejected ones. The KL penalty (controlled by \(\beta\)) slows this down but does not prevent it.

Azar et al. formalised this: DPO's use of the Bradley-Terry likelihood is equivalent to fitting a specific pointwise reward via logistic regression, and that logistic regression objective has no bounded optimal solution when classes are linearly separable, which deterministic preferences always are on training data.

The IPO fix: swap the sigmoid for the identity

The paper introduces a general framework called \(\Psi\)PO. The idea is to optimise directly on pairwise preference gaps without converting them to pointwise rewards first. The general objective is:

where \(\Psi\) is a monotone transform applied to the reward margin, and the target value \(\frac{1}{2}\) encodes the preference distribution (a probability of \(\frac{1}{2}\) of \(y^+\) being preferred corresponds to indifference; a probability of 1 to certain preference).

When \(\Psi = \sigma\) (sigmoid), you recover something close to DPO's loss. When \(\Psi = \text{Id}\) (identity), you get IPO:

This is a squared-error regression loss. It has a unique, bounded optimum: the policy where the reward margin equals \(\frac{1}{2\beta}\). The gradient does not push the margin to infinity; it pulls it toward a target value. Rejected completions are penalised, but never collapsed.

The change in practice is minimal:

# DPO (sigmoid, from TRL)
trainer = DPOTrainer(model, args=DPOConfig(loss_type="sigmoid"), ...)

# IPO (identity transform)
trainer = DPOTrainer(model, args=DPOConfig(loss_type="ipo"), ...)

The dataset format, training loop, and reference model are unchanged.

What changes and what stays the same

The hyperparameter \(\beta\) carries a different interpretation in IPO. In DPO, larger \(\beta\) keeps the policy closer to the reference (stronger KL penalty). In IPO, larger \(\beta\) sets a smaller target margin (\(1/2\beta\)), meaning the policy is encouraged to be less confident about its preference rankings. The optimal \(\beta\) for DPO does not transfer directly to IPO; expect to re-tune.

When it falls down

IPO is not immune to distribution shift. The squared-error formulation has a bounded optimum given the training distribution, but if the labelled pairs are not representative of the prompts the model will encounter at inference, the estimated preference margins are still unreliable. The fix addresses the within-distribution overfitting problem, not the generalisation problem.

The \(\frac{1}{2\beta}\) target is a crude proxy for the true preference gap. The derivation assumes that the expected preference probability for every pair in the dataset is the same value (roughly 0.75 under standard settings). Real preference data has graded signal: some pairs are nearly tied, others are obvious. Treating all pairs with the same target margin discards that gradient.

Small datasets still suffer, just less catastrophically. With very few preference pairs, even the IPO loss can be tuned to unusual optima because the empirical expectation is a poor estimate of the true expectation. IPO reduces but does not eliminate the need for sufficient labelled data.

Mixing IPO with length normalisation requires care. IPO's reward margin \(h_\theta\) sums log-probabilities over full sequences, which creates the same length bias as DPO: longer completions accumulate larger log-probability mass. If you normalise by sequence length before computing the margin (as in SimPO), the IPO objective no longer has the same theoretical guarantees because the normalisation changes the optimisation landscape.

Empirical results are mixed. On some benchmarks IPO outperforms DPO; on others the difference is negligible or reversed. The theoretical advantage is clearest on small, high-quality datasets where DPO genuinely overfits. On large-scale data with diverse labellers, DPO's implicit Bradley-Terry assumption is less harmful and the sigmoid loss's smoother gradients sometimes help optimisation.

Further reading

• Azar, M. et al. (2023). "A General Theoretical Paradigm to Understand Learning from Human Preferences." arxiv.org/abs/2310.12036 - the IPO paper; read sections 3 and 4 for the \(\Psi\)PO derivation.
• Rafailov, R. et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arxiv.org/abs/2305.18290 - DPO original; sections 4-5 contain the Bradley-Terry derivation that IPO supersedes.
• TRL DPO Trainer docs, loss_type="ipo" entry. huggingface.co/docs/trl/en/dpo_trainer - one-line switch and the full list of alternative loss types implemented in the library.