RLVR: RL from Verifiable Rewards

The central bottleneck in standard RLHF is not the RL algorithm; it is the reward model. A learned reward model is itself a neural network trained on human preferences, which means it is wrong in subtle ways and can be gamed. The moment a policy learns to produce outputs that fool the reward model rather than outputs that are actually good, you have entered the regime Gao et al. (2022) call "reward over-optimisation." Their empirical study found that proxy reward keeps climbing while gold reward plateaus and then declines - a textbook instance of Goodhart's law.

RLVR sidesteps this by switching the reward signal source. Instead of asking "what does the reward model score this output?", you ask "is the final answer correct?" For domains where a ground-truth answer exists and can be checked automatically - maths problems, coding tasks, formal proofs, factual look-ups with known answers - you get a reward signal that is binary, cheap, and essentially unhackable at the semantic level. No proxy model, no preference labels, no drift from the gold standard.

What Makes a Reward "Verifiable"

A verifiable reward has three properties:

This is a strict subset of all possible reward signals. RLVR applies well to maths, competitive coding, SQL generation, unit-tested software, and theorem proving. It does not apply straightforwardly to open-ended generation tasks like summarisation or dialogue, where "correct" is not well-defined - those still need a learned reward model or human preference data.

The verification function v(y, y*) is usually trivially simple: string normalisation followed by exact match, or executing code against a test suite. Lightman et al. (2023) showed that even a coarse outcome-level signal (did the model get the final answer right?) is a powerful training driver, though they also demonstrated that step-level (process) supervision can be stronger when available.

GRPO: The RL Algorithm Behind DeepSeek-R1

The DeepSeek-R1 paper (2025) demonstrated that a language model can develop sophisticated chain-of-thought reasoning through pure RL on verifiable rewards, without any supervised fine-tuning on human-labelled reasoning traces. The RL algorithm they used is Group Relative Policy Optimisation (GRPO), first introduced in DeepSeekMath (Shao et al., 2024) as a memory-efficient alternative to PPO.

In standard PPO for language models, you need four models in memory: the policy, the reference policy (for KL regularisation), a critic (value function), and the reward model. GRPO eliminates the critic entirely.

The trick is to estimate advantages group-relative rather than with a learned value function. For each prompt x, sample a group of G responses {y_1, ..., y_G} and assign rewards {r_1, ..., r_G}. The advantage for response y_i is computed within-group:

A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)

This normalisation is cheap and removes the need for a separate baseline network. The policy gradient update then maximises:

L_GRPO = E[ (1/G) * sum_i min(
    (pi_theta(y_i|x) / pi_old(y_i|x)) * A_i,
    clip(pi_theta/pi_old, 1-eps, 1+eps) * A_i
) - beta * KL(pi_theta || pi_ref) ]

The KL term keeps the policy from drifting too far from the reference (the pre-RL checkpoint). beta is a scalar that controls this penalty. When the reward is binary from a verifier rather than a continuous scalar from a learned model, the policy gradient signal is sparser but far more reliable.

In practice, the within-group normalisation also serves as a form of curriculum signal: when all G responses are wrong (all r_i = 0), all advantages are zero and the gradient is near-zero - the model effectively says "I have no information here." When only some responses are correct, the advantage differentiates good from bad rollouts sharply.

Why Binary Rewards Work Better Than You'd Expect

The intuition against sparse binary rewards is that gradients vanish when the model almost never succeeds. RLVR practitioners handle this through three mechanisms:

1. Temperature sampling during rollout. Rolling out at temperature > 1 increases diversity in the sampled group, ensuring at least some correct answers appear in the group even for hard problems. If no rollout in a group is correct, you get no gradient (the within-group normalisation collapses). This means hard problems simply produce no update, rather than a misleading one - arguably better than a noisy dense reward.

2. Curriculum via dataset difficulty. Starting with problems where the base model already succeeds ~20-40% of the time ensures early training signal, then difficulty is increased as the model improves. This is analogous to the self-paced learning literature.

3. Format rewards alongside correctness. DeepSeek-R1-Zero supplemented the binary correctness signal with a small format reward (the model should put its answer in a <answer> tag). This is still verifiable (rule-based) and guides the model to produce parseable outputs without polluting the semantic reward.

The DeepSeek-R1 paper reports that models trained this way spontaneously exhibit "aha moments" - they allocate more thinking tokens to hard problems and develop self-correction behaviour. These patterns emerge from the reward signal alone, without any explicit supervision on reasoning style.

The KL-Regularised Objective and Why It Matters

Without the KL penalty, RL can push the policy into a mode collapse where it generates gibberish that happens to match the verifier. Even with a clean binary reward, a sufficiently aggressive optimiser will find degenerate solutions.

The KL divergence KL(pi_theta || pi_ref) penalises the policy for drifting from the reference. This serves two purposes: it keeps generations fluent and coherent (since the reference is a competent language model), and it prevents the policy from exploiting corner cases in the verification function.

The beta hyperparameter is critical. Too large, and the model cannot learn anything new - the KL penalty dominates and updates are suppressed. Too small, and the model drifts into incoherent regions. Typical values in published work range from 0.01 to 0.1 for the KL coefficient, but this is highly sensitive to the reward scale (a binary 0/1 reward versus a scaled score shifts the sweet spot considerably).

A subtlety: DeepSeek-R1-Zero (the pure-RL version without any SFT warmup) used the pretrained base model as the reference, which means the KL penalty was anchored at a model with no instruction-following ability. This made KL regularisation both more important (to prevent total collapse of coherence) and harder to tune.

When It Falls Down

Reward hacking via verifier bugs. The verifier is simpler than a learned reward model, but not immune to gaming. If the verifier normalises answers with regex and the model learns to format numbers in a way that forces a match (for example, 0.50 matching 0.5), it can score points without real competence. Code execution verifiers face adversarial inputs that cause unexpected behaviour. The closer the verifier is to a full semantic checker, the safer you are - but also the more expensive.

No signal for open-ended tasks. RLVR is inapplicable when there is no verifiable ground truth. Summarisation, creative writing, and most dialogue tasks cannot be scored with a deterministic function. You are forced back to a learned reward model for these, with all its attendant problems.

Distribution collapse under sparse rewards. When problem difficulty outpaces model ability, the within-group advantage is always zero and training stalls. Unlike PPO with a dense reward, there is no gradient to follow to harder problems. This makes curriculum design load-bearing in a way it is not in RLHF.

KL-collapse vs. reward-collapse tension. As beta is tuned down to improve learning speed, the model can drift into fluent but deceptive outputs that match the verifier on the training distribution but fail on slightly rephrased problems. This is a form of generalisation failure that standard held-out evaluation can miss if the test set is too similar to the train set.

Limited to domains with ground truth. RLVR is, by construction, a training method for narrow task types. It is not a general alignment approach. Building a full assistant requires combining RLVR (for verifiable skills) with preference-based methods (for open-ended behaviour) - as DeepSeek-R1 itself does in its later stages, which add SFT on chain-of-thought traces after the pure-RL phase.

Further Reading

• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948 - The paper that demonstrated large-scale RLVR with GRPO, showing emergent chain-of-thought reasoning from pure RL on verifiable rewards.
• Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arxiv.org/abs/2402.03300 - Introduces GRPO and establishes the maths-reasoning baseline that RLVR methods build on.
• Gao et al. (2022). Scaling Laws for Reward Model Overoptimization. arxiv.org/abs/2210.10760 - Quantifies the reward over-optimisation phenomenon and the role of KL regularisation in controlling it.
• Lightman et al. (2023). Let's Verify Step by Step. arxiv.org/abs/2305.20050 - Compares outcome vs. process supervision for maths, directly relevant to reward signal design in RLVR.