Reward Models as Learned Rewards

When OpenAI released InstructGPT in 2022, a 1.3 billion parameter model outperformed the raw 175 billion parameter GPT-3 on human preference evaluations. The size gap is a distraction; the real story is the reward signal. The model was not trained on a fixed objective like next-token cross-entropy. It was trained to maximise a score produced by a second neural network, one that had learned, from roughly 50,000 human comparisons, what "a good answer" feels like. That second network is the reward model, and understanding it is central to understanding modern LLM alignment.

What a reward model actually is

A reward model (RM) is a language model with its final unembedding head replaced by a single linear layer projecting to a scalar. Given a prompt \(x\) and a completion \(y\), it outputs \(r_\theta(x, y) \in \mathbb{R}\). Nothing about the architecture forces the output to mean anything useful; that meaning is instilled by training.

Training uses a preference dataset of triples \((x, y_w, y_l)\), where \(y_w\) is the human-preferred completion and \(y_l\) the dispreferred one. The Bradley-Terry model gives the probability that \(y_w\) is preferred as:

The training loss is the negative log-likelihood of the observed human choices:

This is structurally identical to a binary classification loss. The RM learns to assign higher scores to responses humans preferred, without ever receiving a direct label of "good" or "bad" - only relative comparisons.

Initialisation matters. The RM is typically initialised from the same pretrained (or SFT) checkpoint used for the policy. This is not just engineering convenience: a model that already understands language can use its representations to reason about quality, helpfulness, and factual consistency. Starting from a randomly initialised network would require far more comparison data to achieve the same quality.

From pairwise comparisons to a policy

Once the RM is trained, the RLHF pipeline uses it as the reward function for a policy gradient update. The policy \(\pi_\theta\) (the language model being aligned) generates a completion \(y\) for prompt \(x\), the RM scores it, and the score becomes the reward in a Markov decision process where each token is an action.

The objective is not to maximise \(r_\theta\) naively. Raw maximisation would immediately exploit the RM, producing unnatural text that scores high but satisfies nobody. The standard fix is a KL-regularised objective:

Here \(\pi_\mathrm{ref}\) is the frozen SFT model, and \(\beta\) is a hyperparameter controlling how far the policy can drift. Intuitively, the KL term is a leash: every bit of reward gained by deviating from the reference distribution costs \(\beta\) nats of divergence. This prevents the policy from collapsing into degenerate high-scoring outputs while still improving on the base model.

In practice, PPO (Proximal Policy Optimisation) is used to maximise this objective. The RM provides the scalar reward at the end of the sequence, and a learned value function provides token-level advantage estimates via generalised advantage estimation.

The proxy gap: what the RM is actually measuring

The RM is not measuring the thing you ultimately want. It is measuring what your human annotators, working under your labelling guidelines, within their shifts, with their individual biases, preferred to click. This gap between the proxy reward and the true objective is the central tension in RLHF.

Gao, Schulman, and Hilton (2022) quantified this precisely. In a controlled experiment with a synthetic "gold" reward model serving as ground truth, they optimised a proxy RM and measured gold reward as a function of KL divergence from the initial policy. The findings:

The RL curve falling below baseline is overoptimisation, sometimes called reward hacking. The policy has found inputs that the proxy RM scores highly but the gold reward does not. The proxy is a learned approximation; RL is very good at finding its holes.

The functional form of overoptimisation follows a roughly \(\sqrt{\text{KL}}\) relationship for best-of-n and a more complex shape for RL. This gives practitioners rough rules: keep the KL budget modest (\(\beta\) around 0.01 to 0.1 in most published systems), and monitor gold-side metrics (human evals, held-out benchmarks) separately from RM score.

When it falls down

Distribution shift. The RM is trained on comparisons from the SFT policy's output distribution. As RL training proceeds, the policy shifts, and the inputs to the RM gradually move out-of-distribution. RM scores become less calibrated precisely when the policy is most different from what generated the training data. One mitigation is iterative RLHF: periodically collect new comparisons on the current policy and retrain the RM.

Annotator disagreement and sycophancy bias. Human raters tend to prefer longer, more confident-sounding, more flattering responses, independent of factual accuracy. An RM trained on such data will score verbose sycophancy highly. This is not a labelling error in the traditional sense; it is an accurate reflection of the elicited preference. The problem is that "what annotators prefer in a 30-second evaluation" diverges from "what is actually useful or true." Careful guidelines, annotator calibration, and targeted adversarial prompts in the comparison set are partial mitigations.

Reward model capacity. A small RM will fail to generalise across diverse prompt types. Llama 2's reward model was the same 70B scale as the policy, which is expensive. Smaller RMs often exhibit high variance on out-of-domain prompts, amplifying overoptimisation risk. Ensembling (training multiple RMs on disjoint data splits and averaging scores) helps at the cost of 2-4x training compute.

Verbosity and format hacking. Without explicit length penalties in the RM loss, policies learn to produce longer outputs unconditionally. The RM has picked up a spurious correlation between length and human preference ratings. Length-normalisation, position-bias corrections, and explicit format penalties in comparison guidelines are all used in practice but are imperfect.

The fundamental limit: human labels are not ground truth. The RM is only as good as the comparisons it was trained on. If the underlying human preferences are inconsistent, easily fooled by confident-sounding falsehoods, or biased by demographic or cultural factors, the RM faithfully encodes those flaws. Scalable oversight approaches (debate, AI-assisted critique, process-reward models) are active research directions aimed at this root problem.

Further reading

• Ouyang et al. (2022), "Training language models to follow instructions with human feedback" (InstructGPT): https://arxiv.org/abs/2203.02155
• Gao, Schulman, Hilton (2022), "Scaling Laws for Reward Model Overoptimization": https://arxiv.org/abs/2210.10760
• Stiennon et al. (2020), "Learning to summarize from human feedback" (the paper that first showed systematic RM-based RL for language): https://arxiv.org/abs/2009.01325
• Christiano et al. (2017), "Deep reinforcement learning from human preferences" (the foundational RL-from-comparisons framework): https://arxiv.org/abs/1706.03741