GRPO: Group Relative Policy Optimisation

Training DeepSeekMath 7B to 51.7% on the MATH competition benchmark without any external tools required something that standard PPO could not deliver cleanly: a policy-gradient update that fits comfortably on the same hardware used for the forward pass. The answer was Group Relative Policy Optimisation (GRPO), introduced by Shao et al. (2024) and subsequently scaled in DeepSeek-R1 to drive the emergent reasoning behaviours that surprised the research community in early 2025.

The core tension GRPO resolves is this: PPO needs a value network (the critic) to compute baselines for variance reduction. For a 7B-parameter policy, training an equally-sized critic alongside it roughly doubles peak memory. GRPO sidesteps this entirely by sampling a group of responses from the current policy for every prompt and computing relative advantages within that group. No separate network; no billion-parameter value head.

The PPO Baseline Problem

Recall the standard policy-gradient update. For a token \(t\) in output \(o\), the gradient scales with the advantage \(A_t = Q(s_t, a_t) - V(s_t)\). The \(V(s_t)\) term is a baseline that reduces variance without introducing bias. In PPO, a learned critic approximates \(V\). Training the critic well requires:

1. A full forward-and-backward pass through a large network.
2. A carefully calibrated value loss (often clipped separately).
3. Memory for the critic's parameters and its optimizer state.

For language models, the "state" is the entire token prefix, so the critic is typically initialised from a copy of the policy and fine-tuned in lockstep. At 7B parameters with Adam states, that is roughly 56 GB extra at bf16 just for the critic, before activations. The memory wall is real.

How GRPO Works

For each training prompt \(q\), GRPO samples a group of \(G\) outputs \(\{o_1, o_2, \ldots, o_G\}\) from the current (old) policy \(\pi_{\theta_{\text{old}}}\). A reward model (or rule-based verifier) scores each: \(\{r_1, r_2, \ldots, r_G\}\).

Advantage computation. Under outcome supervision, the baseline is simply the group mean. The advantage assigned to every token in output \(o_i\) is:

This is group normalisation applied to scalar rewards. No critic needed; the group itself supplies the counterfactual signal ("was this response better or worse than what else the policy would have produced?").

Under process supervision (step-level rewards \(r_i^{(j)}\) for each reasoning step \(j\)), the advantage at token \(t\) accumulates future step rewards, giving finer credit assignment within a chain-of-thought.

The objective. GRPO maximises:

where \(\rho_{i,t} = \pi_\theta(a_t | s_t) / \pi_{\theta_{\text{old}}}(a_t | s_t)\) is the importance-sampling ratio (same as PPO-clip), and \(\beta\) is the KL penalty coefficient.

KL treatment. In standard RLHF with PPO, the KL penalty is added token-by-token to the reward before computing advantages, which contaminates the advantage signal. GRPO instead subtracts the KL term directly from the loss, keeping advantages clean. The KL is estimated with the unbiased approximation \(\mathbb{D}_{\mathrm{KL}}[\pi_\theta \| \pi_{\text{ref}}] \approx \log\frac{\pi_\theta}{\pi_{\text{ref}}} - \left(\frac{\pi_\theta}{\pi_{\text{ref}}} - 1\right)\), which avoids a separate reference-model forward pass just for reward shaping.

Pseudocode sketch:

for batch in dataloader:
    prompts = batch["prompts"]
    # Sample G completions per prompt from the frozen old policy
    outputs = [old_policy.sample(p, n=G) for p in prompts]
    rewards = reward_model.score(outputs)           # shape: (B, G)

    mean_r = rewards.mean(dim=-1, keepdim=True)
    std_r  = rewards.std(dim=-1, keepdim=True) + 1e-8
    advantages = (rewards - mean_r) / std_r         # group-normalised

    # Standard PPO-clip update with advantages above;
    # KL subtracted from loss, not folded into rewards
    loss = ppo_clip_loss(policy, old_policy, outputs, advantages)
    loss += beta * kl_divergence(policy, ref_policy, outputs)
    loss.backward()

GRPO in DeepSeek-R1 and RLVR

DeepSeek-R1 (DeepSeek-AI, 2025) applied GRPO at scale under the banner of Reinforcement Learning with Verifiable Rewards (RLVR). The key insight is that mathematics and competitive programming admit ground-truth reward signals: a symbolic checker verifies a final answer, a compiler runs test cases. These rule-based rewards are cheap, noise-free, and impossible to game through surface-level pattern matching.

DeepSeek-R1 used three reward types:

Crucially, DeepSeek-R1-Zero (the pure-RL baseline, no supervised warm-up) was trained with no neural reward model at all. The authors explicitly noted that neural reward models "may suffer from reward hacking in the large-scale reinforcement learning process," requiring costly retraining cycles. Rule-based RLVR avoids this; the reward signal cannot be exploited because the verifier is deterministic.

The result was emergent behaviour: self-verification, backtracking, and extended chain-of-thought without any human-labelled reasoning traces. The model learned to think longer on harder problems simply because longer correct reasoning chains received higher rewards.

Comparing GRPO to PPO

The memory saving is the headline benefit, but there is a subtlety: generating \(G\) completions per prompt at training time is not free. For \(G = 8\) and a 512-token output, you are computing 8x the decoding compute per step. In practice, this is offset by the elimination of critic gradient computation and by the fact that rollout can run at inference speed (no gradient tape needed).

When It Falls Down

Reward signal quality. GRPO's advantages are only as informative as the reward. With a noisy learned reward model, group normalisation amplifies noise: if all \(G\) outputs receive nearly the same score, the advantages collapse toward zero and gradients vanish. RLVR sidesteps this by using verifiable rewards, but that limits applicability to domains with ground-truth checks.

Reward over-optimisation. Even with verifiable rewards, format rewards and secondary signals (like language consistency) can be gamed. Gao et al. (2022) showed that the relationship between proxy reward and true performance follows Goodhart's law: optimise the proxy hard enough and true performance degrades. GRPO's KL penalty is the primary guard, but selecting \(\beta\) is finicky; too small and the policy drifts, too large and learning stalls.

Group size sensitivity. With small \(G\) (e.g., 2 or 4), variance in the advantage estimates is high. The group-mean baseline only works well when the group is diverse enough to span a meaningful reward range. For problems the policy solves consistently (reward \(\approx 1\) for all outputs) or consistently fails (reward \(\approx 0\) for all), the advantage signal disappears entirely.

Distribution shift. GRPO, like PPO, uses importance sampling to correct for the gap between \(\pi_{\theta_{\text{old}}}\) and \(\pi_\theta\). The clipping ratio \(\varepsilon\) controls this, but stale rollouts (many gradient steps between rollout collection and policy update) can exceed the valid importance-sampling regime.

No process-level signal by default. Outcome supervision gives every token in a response the same advantage. A 512-token chain-of-thought that reaches the right answer by a slightly wrong path gets positive reinforcement on all tokens, including the wrong steps. Process supervision requires dense step-level reward labels, which are expensive to obtain or generate.

Further Reading

• Shao et al. (2024), "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models": https://arxiv.org/abs/2402.03300 - original GRPO paper with full algorithm derivation.
• DeepSeek-AI (2025), "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning": https://arxiv.org/abs/2501.12948 - large-scale application of GRPO with RLVR and emergent reasoning results.
• Gao, Schulman, Hilton (2022), "Scaling Laws for Reward Model Overoptimization": https://arxiv.org/abs/2210.10760 - empirical characterisation of Goodhart's law in RLHF, directly relevant to GRPO's reward design choices.
• Schulman et al. (2017), "Proximal Policy Optimization Algorithms": https://arxiv.org/abs/1707.06347 - foundational PPO paper; understanding the clipped objective and critic role is prerequisite to appreciating GRPO's design.