DPO in Practice

The problem with PPO-based RLHF is not theoretical: it is operational. You need to train a reward model, keep a frozen reference policy in GPU memory alongside the live policy, sample from the policy during training, run a KL-penalised RL update, and tune at least four hyperparameters that interact badly. For a 70B model this translates to weeks of engineering before you see a single useful gradient. DPO (Direct Preference Optimisation, Rafailov et al. 2023) collapses that entire pipeline into a single binary cross-entropy pass over preference pairs.

The Maths in One Screen

Standard RLHF maximises a KL-penalised reward objective:

max_π  E[r(x, y)] - β · KL[π(·|x) || π_ref(·|x)]

Solving this analytically yields the optimal policy in closed form:

π*(y|x)  ∝  π_ref(y|x) · exp( r(x, y) / β )

Rafailov et al. inverted this: instead of training r first and then optimising π*, they expressed r in terms of the log-ratio between policy and reference, then substituted back into the Bradley-Terry preference model. The reward disappears from the computation graph entirely, leaving a loss that depends only on the log-probabilities the model assigns to each response:

L_DPO(θ) = -E_{(x, y+, y-)} [
  log σ(
    β · (log π_θ(y+|x) / π_ref(y+|x))
      - β · (log π_θ(y-|x) / π_ref(y-|x))
  )
]

where σ is the sigmoid, y+ is the preferred response, y- is the rejected one, and β controls how tightly the policy is anchored to the reference. A large β (0.5 and above) keeps the policy close to the reference; a small β (0.01-0.05) allows larger deviation.

In practice, the loss presses the model to widen the implicit reward margin between y+ and y- relative to how the reference model scores them. Training metrics worth watching:

Setting Up a DPO Run with TRL

TRL's DPOTrainer is the standard entry point. Each dataset example needs three fields: a prompt, a chosen completion, and a rejected completion. Conversational format works natively; the trainer applies the chat template automatically.

from trl import DPOConfig, DPOTrainer
from datasets import load_dataset
from peft import LoraConfig

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    ref_model=None,           # uses initial model weights as reference
    args=DPOConfig(
        beta=0.1,             # KL penalty weight
        loss_type="sigmoid",  # standard DPO
        learning_rate=1e-6,   # lower than SFT; DPO is sensitive here
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        max_length=1024,
        bf16=True,
    ),
    train_dataset=dataset,
    peft_config=LoraConfig(r=64, lora_alpha=128, target_modules="all-linear"),
)
trainer.train()

When ref_model=None, TRL freezes a copy of the initial weights as the reference. When training with PEFT/LoRA, the frozen base is the reference and only the adapter is updated, which halves memory overhead compared to holding two separate full models.

Key hyperparameter intuitions:

• beta: Treat it like a regularisation coefficient. Too small (< 0.01) and the policy drifts far from reference, often producing verbose, repetitive text. Too large (> 0.5) and training barely moves.
• learning rate: DPO is considerably more sensitive than SFT. Start at 1e-6 and halve if rewards/margins oscillates.
• label smoothing: Adding a small value (0.1-0.2) via label_smoothing in DPOConfig smooths noisy preference labels and reduces overfitting, particularly on small datasets.

The DPO Loss Zoo

The original sigmoid loss is not the only option. TRL exposes a loss_type parameter covering a growing family of variants, each addressing a specific identified weakness:

IPO (arxiv.org/abs/2310.12036) is worth highlighting. The sigmoid in standard DPO can saturate: once the margin is comfortably positive, the gradient nearly vanishes and the model stops learning from those examples. IPO replaces the sigmoid with an identity mapping, ensuring gradients flow even on easy pairs. Empirically it can improve performance on datasets where many preference pairs are near-identical in quality.

Data Quality is the Dominant Factor

The preference dataset is where most practitioners underinvest. DPO is performing maximum-likelihood estimation over human (or AI) judgements, so annotation noise flows directly into the policy. Practical observations:

• Pairs where the chosen and rejected responses are very similar (both are good, or both are bad) contribute near-zero gradient under standard DPO and mostly add noise. Filter pairs below a reward-gap threshold if you have access to a reward model.
• Off-policy data (pairs generated by a model other than the one being trained) can cause a distributional mismatch: the reference policy assigns very low probability to responses it never would have produced, making the log-ratio numerically unstable. Use iterative DPO (online DPO) or RSO-style rejection sampling to generate pairs from the current policy.
• Length bias is subtle but consistent. If annotators or AI judges systematically prefer longer responses, y+ will tend to be longer, and the model learns to produce verbose output. SimPO's length normalisation (loss_type="sigmoid_norm") addresses this directly.

When It Falls Down

Reference model mismatch. DPO is relative: it measures how much the trained policy diverges from the reference on y+ versus y-. If the SFT checkpoint used as reference is weak (undertrained, out-of-distribution), the log-ratios are noisy and the signal is weak. The post-SFT checkpoint quality gates DPO quality.

Forgetting SFT capabilities. DPO can suppress log-probabilities on y- aggressively, sometimes dragging down probability on related but acceptable completions. Monitor validation perplexity on a held-out SFT dataset alongside DPO reward metrics. A hybrid loss combining DPO with a small SFT term (loss_type=["sigmoid", "sft"] with loss_weights=[1.0, 0.1]) can arrest this.

No online exploration. Classic DPO is fully offline: it trains on a fixed preference dataset collected before training starts. The optimal policy derived by RLHF theory should be sampling from the current policy to generate pairs, not from a static offline collection. In practice this means DPO often underperforms PPO-based RLHF on tasks requiring diversity or exploration (complex reasoning, code generation with many valid solutions). Iterative or online DPO variants close some of this gap but add back operational complexity.

Hyperparameter fragility. Unlike SFT where a 2x change in learning rate just trains slower, DPO can catastrophically collapse: reward margins invert, accuracies drop below 0.5, and the model begins preferring rejected responses. Warm up carefully, log rewards/margins every 10 steps, and checkpoint frequently.

Evaluation is hard. DPO reward accuracy on the training distribution is not a reliable proxy for downstream quality. Models that score high on rewards/accuracies can still produce worse outputs on open-ended benchmarks. Always run a separate held-out evaluation (win-rate against a reference model using an LLM judge, or MT-Bench style ratings) before declaring success.

Further Reading

• Rafailov, R. et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." NeurIPS 2023. https://arxiv.org/abs/2305.18290
• Azar, M. G. et al. (2023). "A General Theoretical Paradigm to Understand Learning from Human Feedback." (IPO / PsiPO). https://arxiv.org/abs/2310.12036
• Ethayarajh, K. et al. (2024). "KTO: Model Alignment as Prospect Theoretic Optimisation." ICML 2024. https://arxiv.org/abs/2402.01306
• HuggingFace TRL DPO Trainer documentation (covers all loss types, PEFT integration, data formats). https://huggingface.co/docs/trl/en/dpo_trainer