Online vs Offline Preference Optimisation

Training on yesterday's model outputs to steer today's model is a logical contradiction, yet that is precisely what most practitioners do when they run DPO on a static preference dataset. The resulting distribution mismatch is not a minor nuisance; Tang et al. (2024) showed experimentally that the performance gap between online and offline alignment methods persists even when model scale is increased, and that offline-trained policies actually become better at pairwise classification while degrading at generation - a pattern that no amount of offline data engineering can fully fix.

The Core Distinction

Offline preference optimisation collects a dataset of (prompt, chosen, rejected) triples once, typically from a fixed snapshot of some SFT model, and then trains the policy against those static labels. DPO (Rafailov et al., 2023) is the canonical example. The Bradley-Terry loss it optimises is:

L_DPO(θ) = -E_{(x, y_w, y_l) ~ D} [
    log σ( β · log π_θ(y_w|x)/π_ref(y_w|x)
           - β · log π_θ(y_l|x)/π_ref(y_l|x) )
]

Here D is fixed at collection time. As π_θ moves away from the SFT policy during training, the ratio π_θ(y|x)/π_ref(y|x) drifts, yet the labels never update to reflect what the current model actually generates. You are fitting a moving target with a stationary rubber band.

Online preference optimisation instead samples two candidate responses from π_θ at each training step, obtains a preference signal (from a reward model, an LLM judge, or humans), and updates the policy immediately on that fresh data. RLHF with PPO is the oldest example; online DPO variants (Guo et al., 2024; Xiong et al., 2023) decouple this from the PPO machinery while preserving on-policy sampling.

The practical taxonomy looks like this:

Iterative methods - re-collecting preference data every N steps - sit between the two extremes and often represent the best practical tradeoff.

Why Distribution Shift Matters

DPO's loss implicitly assumes that the training distribution is close to the current policy's distribution. When the policy has moved, the implicit reward signal the loss assigns to un-seen (prompt, response) pairs can be wildly miscalibrated.

Concretely: suppose the SFT model occasionally produces a verbose, rambling answer. The preference data marks that style as "rejected." After several DPO gradient steps, the model has learned to suppress verbosity, so it almost never produces verbose outputs. But the rejected samples still appear in every batch, providing a gradient signal on a response type the model no longer generates - signal that is now noise relative to the actual current policy's failure modes.

Xiong et al. (2023) formalise this through a KL-constrained contextual bandit lens and show that offline DPO lacks "strategic exploration": it cannot discover that the current policy has developed new failure modes not represented in the original dataset.

Online Methods in Practice

Online RLHF with PPO adds significant complexity: you maintain a policy model, a reference model, a reward model, and a value network simultaneously, while managing KL penalties to prevent reward hacking. That complexity is a real engineering cost, which motivated simpler online variants.

Online AI Feedback (OAIF) (Guo et al., 2024): at each step, sample two responses from π_θ, call an LLM annotator to judge which is preferred, and run a DPO-style update on the resulting pair. This eliminates the need for a separately trained reward model while preserving on-policy data. Human evaluations in that paper show OAIF outperforms both offline DPO and classical RLHF baselines.

Iterative DPO (Xiong et al., 2023): collect preferences on the current policy every K steps, fine-tune, repeat. Even a single extra round of collection substantially closes the gap with fully online methods, because the most dangerous distribution mismatch builds up early in training when the policy changes fastest.

The Calandriello et al. (2024) result connecting online IPO with Nash Mirror Descent adds a theoretical underpinning: online preference optimisation is implicitly finding a Nash equilibrium between the policy and a preference oracle, and the offline approximation is cutting off that game after one move.

The Reward Hacking Asymmetry

Reward hacking looks different in the two regimes:

• Offline: the model can exploit statistical quirks in the fixed dataset - response length, surface-level tokens that correlate with the "chosen" label - because nothing corrects for these artifacts as training proceeds. The longer training runs, the more entrenched the exploits become.

• Online: the model and reward model interact in a feedback loop. The policy discovers exploits more aggressively (it has a live oracle to probe), but those exploits become visible in the evaluation loop faster, allowing earlier intervention. PPO's KL penalty is the primary guard rail; tuning β is critical.

Neither regime is immune. Online methods simply surface reward hacking faster and more legibly.

When It Falls Down

Offline DPO on covariate-shifted data deteriorates sharply when the preference dataset was collected from a weaker or very different model than the one being trained. If the chosen/rejected gap is mostly about basic fluency - already solved by the stronger model - the loss provides almost no useful gradient.

Online methods with a weak reward model amplify errors. Because the reward model is queried on the current policy's outputs, any systematic bias (e.g., preferring longer responses) gets incorporated into every gradient step. The offline regime at least has the accidental advantage that stale labels partially decorrelate from the model's current exploitation strategies.

LLM-judge-based online feedback (OAIF and variants) inherits all the biases of the judge model: position bias, verbosity bias, self-preference. These biases propagate directly into policy behaviour because the judge is queried on every sampled pair.

Compute cost: a fully online setup requires inference at training time to generate candidate pairs. On large models (70B+), this is expensive enough that iterative batched approaches are usually chosen over step-level online updates.

Cold-start problem: online methods need a policy good enough to produce non-degenerate outputs before the preference signal is informative. Starting from a weak SFT checkpoint can cause the first few online rounds to produce uninformative or incoherent pairs, wasting reward model queries.

Further Reading

• Rafailov, R. et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv:2305.18290. The foundational offline method whose distribution-shift limitations motivated online variants.
• Tang, Y. et al. (2024). "Understanding the performance gap between online and offline alignment algorithms." arXiv:2405.08448. The systematic empirical analysis showing the gap persists with scale and traces it to discriminative-vs-generative capability divergence.
• Xiong, W. et al. (2023). "Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint." arXiv:2312.11456. Provides unified theory across offline, online, and hybrid regimes; introduces iterative DPO.
• Guo, S. et al. (2024). "Direct Language Model Alignment from Online AI Feedback." arXiv:2402.04792. Demonstrates that replacing the reward model with an online LLM judge outperforms both offline DPO and classical RLHF.