The Infrastructure of LLM RL

Training GPT-3 on next-token prediction cost roughly $4.6 million in compute. InstructGPT, the RLHF-tuned successor that outperformed it on human preference, was trained on a 1.3B-parameter model. The gap between the two is not scale; it is the post-training stack. Understanding why requires looking at what RL for language models actually runs in memory.

The Four-Model Problem

Standard deep RL trains one network. RLHF trains four simultaneously:

All four need to hold a sequence in memory at the same time to compute a single PPO gradient update. For a 7B-parameter policy, a naive setup requires roughly 4x the VRAM of inference alone, before optimizer states. This is the central infrastructure constraint that drives every engineering decision downstream.

The policy and reference model are typically initialised from the same SFT checkpoint. The reward model is trained separately on human preference data - annotators rank completions, then a Bradley-Terry model converts those rankings into scalar scores. The value model is often initialised from the reward model's backbone with a new regression head.

The KL-Regularised Objective

Raw RL against a reward model will quickly degenerate: the policy discovers ways to maximise the RM's score without producing text that is actually useful. To prevent this, the optimisation target is not bare reward but a penalised version:

J(θ) = E[r(x, y)] - β · KL[π_θ(y|x) || π_ref(y|x)]

where: - π_θ is the current policy - π_ref is the frozen reference (SFT) model - β controls how far the policy is allowed to drift - r(x, y) is the reward model's scalar for response y given prompt x

The KL term penalises the policy for assigning meaningfully different probabilities to tokens than the reference model would. At β = 0 the model is free to exploit the reward function without bound. At large β the model barely moves from the SFT baseline. In practice, InstructGPT used β ≈ 0.2, though this is tuned per run and is sensitive to reward model quality.

The KL divergence is computed token-by-token and summed over the full sequence. During a PPO rollout, the log-probability ratio log π_θ / log π_ref is computed for every generated token, making the reference model's forward pass a mandatory cost on every training step.

RLVR: When You Have a Verifiable Signal

Human preference labels are expensive, inconsistent, and slow. For tasks with deterministic correct answers - mathematics, code execution, formal proofs - you can replace the RM entirely with a rule-based verifier. This is called Reinforcement Learning from Verifiable Rewards (RLVR).

DeepSeek-R1 (January 2025) is the clearest public example. The model was trained on maths and coding problems where correctness can be checked programmatically: the answer either matches the ground truth or it does not. There is no reward model to train, no human labelling pipeline, and no risk that the policy will find subtle ways to fool a learned scorer. The reward signal is binary (or near-binary), clean, and cannot be gamed.

This changes the infrastructure meaningfully: the reward model component is replaced by a deterministic function call. The value model can sometimes be removed too (see GRPO below). What remains is the policy, the reference model, and the verifier.

RLVR cannot cover tasks like tone, creativity, or helpfulness, where no ground-truth oracle exists. It is a powerful specialisation, not a general replacement.

GRPO: Removing the Value Model

Group Relative Policy Optimisation (GRPO) was introduced in the DeepSeek-R1 work. The key insight is that the value function is the most expensive component to train and maintain: it requires a separate model of roughly equal size to the policy, and its estimates are often noisy on long-horizon text generation.

GRPO replaces per-token value estimates with a group baseline. For a single prompt, the policy generates a group of G completions (typically 4-16). The reward for each completion is normalised relative to the group mean and standard deviation:

advantage_i = (r_i - mean(r)) / std(r)

This is a control variate: completions that scored above average get positive advantage, below-average completions get negative advantage, and the mean cancels out. No value model is needed because the baseline comes from the policy's own rollouts.

The gradient update then resembles PPO's clipped surrogate objective, but without the critic:

L_GRPO = E[ min(ρ_i · A_i, clip(ρ_i, 1-ε, 1+ε) · A_i) ] - β · KL[π_θ || π_ref]

where ρ_i = π_θ(y_i|x) / π_old(y_i|x) is the probability ratio and A_i is the group-relative advantage.

Memory footprint drops from four models to effectively two-and-a-half (policy + reference + the overhead of holding G completions). For a 70B-parameter model this is the difference between fitting on a single 8-GPU node or not.

The tradeoff is variance: group-relative baselines have higher variance than a trained value function when the group size is small or the reward signal is sparse. Larger groups reduce variance at the cost of more rollout compute.

When It Falls Down

Reward hacking. The reward model is a proxy, not the true objective. Gao, Schulman, and Hilton (2022) showed empirically that as RL optimisation continues, the policy's score on the RM keeps rising while its true human-preferred quality plateaus and then falls - a textbook Goodhart's Law pattern. The KL penalty slows this but does not stop it; once the policy finds a systematic exploit of the RM's blind spots, the KL cost is paid and the hacking persists.

Reference model staleness. The KL penalty is computed against a fixed SFT checkpoint. If the policy drifts far enough, the KL term becomes dominated by tokens that are simply out-of-distribution for the reference, not tokens that are genuinely harmful or unhelpful. The penalty loses its semantic meaning and becomes a wall around the initial checkpoint rather than a principled regulariser.

GRPO variance collapse. When all completions in a group receive the same reward (the verifier returns 1 for all or 0 for all), the group standard deviation is zero and the advantage is undefined. Implementations handle this by skipping the update for that prompt or adding a small epsilon. But it means prompts where the model is either very good or very bad produce no gradient signal, leaving systematic weaknesses unaddressed.

Value model divergence in PPO. The critic must be trained to track moving-target reward statistics as the policy evolves. On long completions (512+ tokens), temporal credit assignment degrades: the value function cannot reliably attribute a bad final reward to the specific token choices that caused it. Early tokens get near-zero gradient signal, which is why reasoning models sometimes produce strong conclusions from weak premises.

VRAM wall at scale. Running four models simultaneously caps the maximum model size that can be trained on a given hardware budget. Most publicly reported RLHF runs on models above 70B use either model parallelism across many nodes (costly and operationally fragile) or approximate the value function with a much smaller model (biasing the gradient).

Further Reading

• Ouyang et al. (2022), "Training language models to follow instructions with human feedback" - the InstructGPT paper that introduced the four-model RLHF setup at scale: https://arxiv.org/abs/2203.02155
• Schulman et al. (2017), "Proximal Policy Optimization Algorithms" - the RL algorithm underlying most LLM post-training: https://arxiv.org/abs/1707.06347
• Gao, Schulman, Hilton (2022), "Scaling Laws for Reward Model Overoptimization" - empirical measurement of reward hacking dynamics: https://arxiv.org/abs/2210.10760
• DeepSeek-AI (2025), "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" - GRPO and RLVR at production scale: https://arxiv.org/abs/2501.12948