Reasoning RL and R1-Style Training

When DeepSeek released R1 in January 2025, the first public shock was the benchmark numbers. The quieter shock, on reading the paper, was the training recipe: no human-annotated reasoning trajectories, no teacher distillation for the RL phase, just a language model generating answers and a binary reward signal saying "correct" or "wrong." Self-reflection and chain-of-thought appeared to emerge from that signal alone. The question this concept addresses is why that works, and what it implies for post-training methodology more broadly.

RLHF Is Already RL

It is worth being precise about vocabulary before the terminology splits into RLHF, RLVR, GRPO, and R1-style training.

Classic RLHF (InstructGPT; Ouyang et al., 2022) is a three-stage pipeline:

1. Supervised fine-tuning (SFT) on high-quality demonstrations.
2. Train a reward model (RM) on pairs of completions ranked by human annotators.
3. Optimise the SFT model against the RM using PPO, subject to a KL penalty anchoring the policy to the SFT reference.

Stage 3 is genuine RL: the policy is the LLM, the action space is token generation, the reward is the RM score, and the KL term is a regulariser that prevents the policy from drifting so far from the reference that the RM is operating out of distribution. The objective is:

J(θ) = E[r_φ(x, y)] - β · KL[π_θ(y|x) || π_ref(y|x)]

where r_φ is the learned reward model, π_θ is the current policy, π_ref is the SFT reference, and β controls how tightly the policy is leashed. A small β lets the policy chase reward aggressively; a large β keeps it close to the SFT distribution.

The human feedback is upstream of stage 3. RL from human feedback is conceptually a two-part system: a reward learning problem (fitting r_φ) and a policy optimisation problem (maximising J). RLHF entangles both; RLVR separates them by removing the learned RM entirely.

RLVR: Replacing the Reward Model with a Verifier

Reinforcement Learning from Verifiable Rewards (RLVR) is the term for training regimes where the reward signal is computable, not learned. For mathematics and coding:

• A maths answer is either numerically correct or not. The verifier is a string-match or symbolic evaluator.
• A code solution either passes the test suite or it doesn't.

Because the reward is ground-truth-verifiable, r_φ collapses to a deterministic function and reward model over-optimisation becomes structurally impossible for the task reward itself. The KL penalty is still present to prevent the policy drifting to degenerate token sequences, but the RM hacking failure mode is eliminated.

The practical implication is stark: you do not need a large, carefully trained reward model. You need a dataset of problems with verifiable answers, a reference policy, and a KL budget.

GRPO: Getting PPO's Benefits Without the Critic

PPO (Proximal Policy Optimisation) requires a value function (critic) trained alongside the policy. For LLMs this is expensive: the critic is typically another large model, doubled memory and compute overhead.

DeepSeekMath (Shao et al., 2024; arXiv:2402.03300) introduced Group Relative Policy Optimisation (GRPO) to sidestep this. Instead of a learned critic estimating the value of a state, GRPO estimates advantages by sampling a group of completions for the same prompt and normalising rewards within the group:

For prompt x, sample G outputs {y_1, ..., y_G}
Compute rewards {r_1, ..., r_G}
Advantage for y_i: A_i = (r_i - mean(r)) / std(r)

The policy gradient update is then:

L_GRPO(θ) = -E[ min(
    ρ_i · A_i,
    clip(ρ_i, 1-ε, 1+ε) · A_i
)] + β · KL[π_θ || π_ref]

where ρ_i = π_θ(y_i|x) / π_θ_old(y_i|x) is the probability ratio (the same PPO clipping mechanism). The KL term is computed token-by-token against the reference policy rather than against the old policy.

GRPO's advantage over PPO in this setting: no critic, half the memory, and advantage estimates that are inherently relative to the difficulty of the specific prompt (normalising within the group controls for prompts that are uniformly easy or hard).

The R1 Training Recipe

DeepSeek-R1 (DeepSeek-AI, 2025; arXiv:2501.12948) applied RLVR plus GRPO to a base language model with a deliberate four-stage curriculum:

The "cold start" insight is pragmatic: a raw base model generates chaotic outputs when given only a sparse reward signal. A handful of long-CoT supervised examples gives it enough structure to produce format-stable trajectories before GRPO takes over.

The emergent behaviours observed during stage 2 were notable. Without being explicitly trained to, the model began generating segments that resembled self-verification ("wait, let me check this"), backtracking on incorrect intermediate steps, and extending chain length when problems were harder. These behaviours correlated with reward improvement, suggesting they are instrumentally discovered strategies rather than imitated ones.

Kimi k1.5 (Kimi Team, 2025; arXiv:2501.12599) followed a similar RLVR approach and reached 77.5 on AIME 2024 and 96.2 on MATH 500. It also explored long-to-short CoT transfer: the RL-trained long-chain policy can distil compressed reasoning into a short-chain model via rejection sampling, recovering inference-efficiency without sacrificing much accuracy.

When It Falls Down

Reward over-optimisation still applies to format rewards. If the verifier rewards a specific answer format (e.g., answers enclosed in \boxed{}), the model can learn to produce correct-format boxes around wrong answers, or game token sequences that look like reasoning but do not contain it. Gao, Schulman, and Hilton (2022; arXiv:2210.10760) characterised the general scaling law for this failure mode: proxy-reward improves monotonically while true reward peaks and then degrades. The KL penalty delays but does not prevent this if β is too small.

Out-of-distribution generalisation is uneven. RLVR on maths and code does not automatically transfer to domains without clean verifiers. Long-chain reasoning on open-ended writing or factual synthesis lacks a ground-truth checker. Applying binary rewards to subjective tasks collapses back to RLHF with all its annotation costs.

Training instability at long sequence lengths. GRPO samples multiple full completions per prompt. For reasoning tasks, those completions can run to thousands of tokens. Memory pressure, gradient variance from high-reward-variance groups, and entropy collapse (the policy assigning near-zero probability to non-greedy tokens) are all reported failure modes in practice.

Cold-start data quality sets the ceiling. The RL phase amplifies tendencies present in the SFT initialisation. Poor cold-start data yields models that explore inefficiently or discover degenerate reasoning shortcuts early. The R1 recipe works partly because the cold-start step was carefully curated.

Reward sparsity on hard problems. When problems are near the policy's capability limit, almost all samples in a GRPO group receive zero reward. The normalised advantages are effectively zero, the gradient vanishes, and training stalls. Curriculum scheduling (ordering problems by difficulty) mitigates this but adds hyperparameter sensitivity.

Further Reading

• DeepSeek-R1 paper: arxiv.org/abs/2501.12948 - the primary reference for R1-style training, cold-start, and emergent reasoning.
• DeepSeekMath and GRPO: arxiv.org/abs/2402.03300 - introduces GRPO and the RLVR framing for mathematical reasoning.
• Kimi k1.5: arxiv.org/abs/2501.12599 - independent validation of scaled RLVR plus long-to-short distillation.
• Reward model overoptimisation: arxiv.org/abs/2210.10760 - Gao, Schulman, Hilton on scaling laws for the proxy-reward / true-reward divergence.