Evaluating RL-Tuned Models

When OpenAI published results for their RLHF-trained summarisation model in 2020, they noted something uncomfortable: a model that scored higher on the learned reward signal produced summaries that humans actually preferred, yet that same reward signal could be over-optimised to produce text that scored high while being hollow. The proxy had become the target, and the target had quietly moved. That tension sits at the heart of evaluating any RL-tuned model.

Why Standard Benchmarks Mislead

Pre-training and supervised fine-tuning (SFT) benchmarks such as MMLU, HellaSwag, or HumanEval are designed to probe raw knowledge and surface-level instruction following. They assume a relatively well-behaved distribution shift between training and evaluation. RL post-training breaks that assumption in at least three ways.

Distribution shift at the output level. RL optimises over the policy's own generated text, not reference completions. After several thousand gradient steps the model's output distribution may sit far from the SFT checkpoint. A benchmark that probes fill-in-the-blank-style accuracy may not sample from the region where the model now lives.

Reward model entanglement. If the benchmark was used to inform reward model training or was included in the preference dataset, then high benchmark accuracy signals contamination rather than capability. This is not hypothetical: several chat-model leaderboards have documented steep drops in score when new, unseen benchmark variants are introduced.

Gaming via verbosity or hedging. RL optimised against a human-preference reward model learns quickly that hedged, verbose answers are rated more favourably, independent of correctness. A model that adds qualifications and filler around a wrong answer can score higher on LLM-as-judge evaluations than a concise, correct competitor. Dubois et al. (2024) quantified this directly for AlpacaEval, showing strong positive correlation between output length and win-rate.

The Evaluation Hierarchy

A practical evaluation stack for RL-tuned models has three layers, each checking a different failure mode.

Layer 1: Verifiable correctness on held-out sets

For domains where ground truth exists (mathematics, code, formal reasoning), use execution-based or proof-checked evaluation. AIME problems, competition-grade coding judges (LiveCodeBench), and formal verification suites give a binary signal: the answer is right or it is not. These benchmarks are hardest to game through verbosity or style because there is no LLM judge to fool.

Critically, the held-out set must be temporally separated from the reward model's training data. Problems released before the model's training cutoff are potential contaminants regardless of whether the developers consciously included them.

Layer 2: Human preference versus reward model preference

The reward model is a proxy for human preferences, not a ground truth. The gap between the two is the over-optimisation gap. Measuring it requires human evaluation on a sample of model outputs after RL training. Gao et al. (2022) showed that this gap follows a predictable scaling law: proxy reward increases monotonically while gold human reward peaks and then declines as KL divergence from the SFT checkpoint grows:

gold_reward ≈ a * sqrt(KL) - b * KL

where a and b are empirical constants that depend on reward model size and dataset size. The implication: evaluators who only report proxy reward scores are reporting a metric that is structurally optimistic.

A practical approximation when human labellers are expensive is to use a held-out reward model (trained on a disjoint preference dataset) as a secondary judge and track the gap between the training reward model and this auditor model over training steps.

Layer 3: Behavioural and alignment probes

Instruction-following, refusal calibration, and sycophancy are not well captured by accuracy benchmarks. MT-Bench and Chatbot Arena (Zheng et al., 2023) use LLM-as-judge and crowdsourced human preference respectively. The key methodological points:

• LLM judges exhibit positional bias (the first response is rated more highly) and self-preferential bias (a GPT-4 judge favours GPT-4 outputs). Both should be controlled for with randomised orderings and cross-judge comparisons.
• Chatbot Arena's Elo-based ranking is more robust to individual judge noise but requires high sample volume and is slow to converge.
• Sycophancy probes (does the model change its answer when the user pushes back without providing new evidence?) are cheap to construct and reveal reward-model gaming that standard benchmarks miss entirely.

Separating RL Gain from SFT Gain

A common confound in published results: the model reported as "RL-tuned" was also trained on a larger or higher-quality SFT dataset than the baseline. The RL contribution to the final evaluation delta is then unclear. Clean ablation requires:

1. A fixed SFT checkpoint used as the starting point for both the RL-tuned model and all baselines.
2. Evaluation at matched KL divergence from that SFT checkpoint (not at fixed training step counts), so that models are compared at equivalent amounts of distribution shift.
3. Reporting of both proxy reward and gold (human or held-out-auditor) reward, not just downstream benchmark scores.

DeepSeek-R1 (2025) is an example where the ablation is relatively transparent: the paper compares pure RL (DeepSeek-R1-Zero) against RL with cold-start SFT data, making the contribution of each stage separately measurable.

Benchmark Contamination in RLVR Settings

Verifiable reward RL (RLVR) setups - where reward is computed from the correctness of a final answer on maths or code problems - are particularly susceptible to a subtle contamination mode: the training problem distribution may overlap with evaluation benchmarks if both are drawn from the same competition archives. LiveCodeBench (2024) was specifically designed to mitigate this by pulling from competitions post-dating most models' training cutoffs.

Even with temporal separation, solution-style contamination can occur: if the model has seen stylistically similar worked solutions during pretraining, it may recognise problem patterns without having memorised exact answers. Detecting this requires canary problems (problems that have never appeared online and were constructed solely for evaluation).

Calibration and Chain-of-Thought Faithfulness

RL training changes calibration in ways that accuracy metrics do not capture. Models trained with outcome reward often become overconfident on the classes of problems the reward model rewarded heavily. On out-of-distribution problems they may produce confident-looking chains of thought that reach wrong answers.

A useful diagnostic: compute Expected Calibration Error (ECE) before and after RL, conditioned on problem difficulty tier. RL-tuned models frequently show degraded calibration at high difficulty even when aggregate accuracy is stable. Additionally, check whether stated reasoning steps actually support the conclusion (chain-of-thought faithfulness), since outcome-based RL has no incentive to produce mechanistically correct intermediate reasoning.

When It Falls Down

Over-reliance on LLM judges. When the evaluation judge is itself a large instruction-tuned model, its biases (length preference, style preference, self-preference) can systematically inflate scores for RL-tuned models that were trained to produce similar outputs. This is a positive feedback loop that makes leaderboards unreliable over time.

KL threshold mismatch. Reporting evaluation results at a single KL value does not reveal whether the model is on the ascending or descending portion of the gold reward curve. A model 10 nats past the peak looks identical in benchmark score to one 10 nats before it, but has genuinely degraded quality.

Narrow verifiable domains. RLVR evaluation looks clean in mathematics and coding, but most real-world tasks do not have a formal verifier. Attempts to use LLM-based graders as pseudo-verifiers reintroduce all the judge-bias problems from Layer 3.

Distribution collapse masking. A model that has collapsed to a narrow output mode (for example, always producing bulleted lists with three points) may score well on individual benchmark items while having lower coverage and diversity than a more capable model. Diversity metrics (self-BLEU, output entropy) should accompany point-accuracy metrics.

Benchmark saturation. Once a benchmark is widely known, future RL training runs are likely to include preference data collected from users discussing those benchmarks. Static benchmarks saturate; evaluation should rotate to include live, continuously updated problem sets.

Further Reading

• Learning to Summarize from Human Feedback - Stiennon et al., NeurIPS 2020
• Scaling Laws for Reward Model Overoptimisation - Gao, Schulman, Hilton, 2022
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - Zheng et al., 2023
• Length-Controlled AlpacaEval: Debiasing Automatic Evaluators - Dubois et al., 2024