Offline RL and the DPO Connection

The 2023 DPO paper opens with a quietly devastating observation: every RLHF practitioner is already solving a constrained optimisation problem that has a known closed-form solution. They just didn't realise it, so they were using PPO instead.

That mismatch explains both the elegance of DPO and its limitations. Understanding the gap between them requires knowing exactly where offline RL ends and where online RL begins.

The objective everyone is actually solving

Standard RLHF maximises expected reward while penalising divergence from a reference policy:

max_π  E_{x~D, y~π} [ r(x, y) ]  -  β · KL[ π(y|x) ‖ π_ref(y|x) ]

Here x is a prompt, y is a completion, r is the reward model, π_ref is the supervised fine-tuned (SFT) baseline, and β controls how far the optimised policy is allowed to drift. This is precisely the KL-regularised RL objective covered in the foundational treatment of RLHF-as-RL.

The key fact, known from the KL-constrained optimisation literature since at least Ziegler et al. (2019), is that this objective admits a closed-form optimal policy:

π*(y|x)  ∝  π_ref(y|x) · exp( r(x, y) / β )

This says: take the reference distribution and re-weight each completion by its exponentiated reward, normalised over all completions. The partition function Z(x) = Σ_y π_ref(y|x) · exp(r(x,y)/β) is intractable to compute directly for autoregressive sequences, which is exactly why practitioners turned to PPO.

DPO's insight is to invert this relationship. If the optimal policy is π*, then the reward implied by π* is:

r(x, y)  =  β · log[ π*(y|x) / π_ref(y|x) ]  +  β · log Z(x)

Plug this reparameterisation into the Bradley-Terry preference model (the standard assumption that human annotators prefer y_w over y_l with probability σ(r(x,y_w) - r(x,y_l))), and log Z(x) cancels out in the difference. The resulting training objective is:

L_DPO(π_θ) = -E_{(x, y_w, y_l) ~ D} [
    log σ( β · log[π_θ(y_w|x) / π_ref(y_w|x)]
           - β · log[π_θ(y_l|x) / π_ref(y_l|x)] )
]

This is a binary cross-entropy loss over preference pairs. No reward model. No rollouts. No PPO update loop. The policy network π_θ is simultaneously the reward model, encoded implicitly in its log-ratio against π_ref.

Why this is offline RL

Offline RL (sometimes called batch RL) refers to any approach that learns a policy entirely from a fixed dataset of transitions collected under some other behaviour policy, with no further environment interaction. The contrast is online RL, where the agent collects new data by acting in the environment during training.

DPO is offline RL in an almost literal sense:

The "offline" character is not incidental to DPO's simplicity. It is the source of both its efficiency and its brittleness. Because π_θ never queries the reward function on its own generations, it cannot learn to handle responses that are not represented in the preference dataset. This is the classic offline RL distributional shift problem, applied to language.

The preference dataset is typically collected by sampling from the SFT policy or an early RLHF checkpoint, then having annotators label which completion they prefer. DPO trains directly on those pairs. If the training distribution has gaps, the learned implicit reward has no way to fill them.

The implicit reward and what it measures

The term β · log[π_θ(y|x) / π_ref(y|x)] is the log-ratio of the fine-tuned policy to the reference. After training, this quantity serves as a reward signal: completions the model has learned to produce more often than the reference tend to have a high implicit reward.

This relationship has a practical diagnostic use. Given a trained DPO model and its reference, you can rank candidate completions by this log-ratio and ask whether the ranking makes sense. If it does not, something went wrong in training, often due to the issues described below.

The connection also clarifies what β does. Large β forces π_θ ≈ π_ref, compressing the implicit reward range and making the model conservative. Small β allows large divergence, risking the same reward over-optimisation pathologies as unconstrained RLHF. DPO does not escape the KL-reward tradeoff; it just parameterises it differently.

Variants and successors

DPO's offline nature prompted several follow-on methods that attempt to recover online data collection:

IPO (Identity Preference Optimisation, Azar et al., 2023) generalises the preference objective beyond Bradley-Terry, showing that DPO's specific loss function depends on this probabilistic model of human choices, which may not always hold.

Online DPO / Iterative DPO collects new preference pairs from the current policy at intervals, then re-runs DPO. This recovers some of the distributional correction that PPO gets continuously, at the cost of periodic human (or AI) labelling. Meta's Llama 2 used iterative RLHF rather than a single offline pass for exactly this reason.

KTO (Kahneman-Tversky Optimisation, Ethayarajh et al., 2024) replaces pairwise preferences with binary desirability signals, observing that collecting pairs is expensive and that DPO's implicit assumptions about the preference model embed cognitive biases from prospect theory. KTO demonstrated competitive performance against DPO at 1B-30B scale without needing pairwise annotations.

None of these eliminate the core offline limitation. They either accept it (KTO), mitigate it with periodic online batches (iterative DPO), or extend the theoretical framework without solving distributional shift (IPO).

When it falls down

Out-of-distribution completions receive arbitrary implicit rewards. If the policy during deployment generates completions unlike anything in the preference dataset, the log-ratio log[π_θ(y|x) / π_ref(y|x)] gives unreliable reward estimates. This is mathematically identical to the extrapolation problem in offline RL noted by Levine et al. in the offline RL survey literature.

Length bias. In practice, DPO training often upweights longer completions. The binary cross-entropy objective does not normalise by sequence length, so the policy can increase the log-ratio for a preferred response simply by extending it, rather than improving quality. Analyses of DPO-trained models frequently show length inflation, a form of reward hacking that the explicit reward model in PPO can be designed to penalise.

Degenerate decreasing of rejected probability. Empirical work (Feng et al., 2024) found that DPO reduces the probability of dispreferred completions faster than it increases the probability of preferred ones. In the extreme case, the policy degrades the reference's capabilities without meaningfully improving on the preferred direction.

Sensitivity to SFT quality. DPO uses π_ref both as the regulariser and as the implicit reward baseline. If the SFT model is weak or misaligned, the reference is a poor anchor. PPO can adapt because the reward model is independent of the policy; DPO cannot, because the reward and the policy are the same object.

Preference dataset quality is load-bearing. PPO can partially survive noisy reward labels because the policy explores and receives corrective signal. In DPO, noisy preferences are baked directly into the loss with no correction mechanism. Studies of large-scale annotation pipelines routinely find inter-annotator agreement below 80% on subtle helpfulness distinctions, a regime where DPO's sensitivity to label quality matters.

Further reading

• Rafailov, R. et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv:2305.18290. The original paper; Sections 3-4 contain the full derivation.
• Ouyang, L. et al. (2022). "Training language models to follow instructions with human feedback." arXiv:2203.02155. The InstructGPT paper that established the KL-regularised RLHF objective at scale.
• Azar, M. G. et al. (2023). "A General Theoretical Paradigm to Understand Learning from Human Preferences." arXiv:2310.12036. Introduces ΨPO, showing DPO and RLHF as special cases of a unified framework.
• Ethayarajh, K. et al. (2024). "KTO: Model Alignment as Prospect Theoretic Optimisation." arXiv:2402.01306. A practical alternative that replaces pairwise preferences with binary signals and connects preference learning to behavioural economics.