RL for Language Models
RLHF as an RL problem, KL-regularised objectives, GRPO, RLVR, and reward over-optimisation.
21concepts
142flashcards
172minutes of reading
- 01 Best-of-N and Inference-Time Selection Best-of-N sampling generates multiple completions from a language model and returns the one ranked highest by a reward model, trading inference compute for quality without updating any weights.
- 02 Length and Format Reward Hacking When a reward model assigns higher scores to longer or more structured responses regardless of quality, RL training exploits that signal and the policy degrades into verbose padding and hollow formatting instead of improving reasoning.
- 03 Process vs Outcome Rewards Process reward models score each reasoning step individually, giving denser training signal than outcome rewards and catching errors before they contaminate a final answer.
- 04 RLHF as a Reinforcement-Learning Problem RLHF recasts language model alignment as a policy-optimisation problem where a reward model trained on human comparisons provides the scalar signal that PPO uses to update the language policy.
- 05 Rejection Sampling as RL Rejection sampling fine-tuning filters a model's own outputs by correctness and trains on the survivors, achieving a policy-improvement step that is mathematically equivalent to one round of RL but without an explicit optimiser loop.
- 06 Reward Models as Learned Rewards A reward model is a classifier trained on human preference comparisons that outputs a scalar score, standing in for the true human utility function during RL fine-tuning of language models.
- 07 Reward Over-Optimisation When a language model is trained too aggressively against a proxy reward model, it learns to exploit the proxy rather than genuinely improve, causing measured reward to climb while true quality declines.
- 08 The Bandit Framing of RLHF RLHF treats a language model as a contextual bandit that receives a single scalar reward per complete response, making full Markov decision process machinery unnecessary but also hiding the dangers of reward over-optimisation.
- 09 Credit Assignment over Long Generations Explains why distributing a single scalar reward back across hundreds of generation steps is the central unsolved tension in RL for language models, and surveys the main strategies used to address it.
- 10 Evaluating RL-Tuned Models Standard NLP benchmarks break silently when applied to RL-tuned models because the training objective optimises for a reward signal that can diverge from genuine capability, requiring a distinct evaluation stack to distinguish real improvement from reward gaming.
- 11 Exploration in Language-Model RL Language-model RL training collapses silently when the policy stops generating diverse completions, and standard RL exploration techniques must be reinterpreted to work inside a token-sequence action space.
- 12 GRPO: Group Relative Policy Optimisation GRPO removes the critic network from PPO by estimating baselines from a sampled group of outputs, halving the GPU footprint while delivering competitive reasoning improvements.
- 13 Multi-Turn and Agentic RL Multi-turn and agentic RL extends single-response RLHF to sequences of actions across environment steps, requiring credit assignment, trajectory-level rewards, and new training algorithms suited to long-horizon tool-using agents.
- 14 Offline RL and the DPO Connection DPO re-derives the standard KL-regularised RLHF objective and solves it in closed form, turning preference alignment into a supervised classification loss over offline data without ever sampling from the policy during training.
- 15 PPO for Language Models Proximal Policy Optimisation clips the policy update ratio to prevent destructive gradient steps, making it the workhorse algorithm for RLHF fine-tuning of large language models.
- 16 RLVR: RL from Verifiable Rewards RLVR replaces the trained reward model in RLHF with an automated verifier that checks correctness against a ground-truth answer, producing a clean binary signal that sidesteps reward hacking and scales to tasks like maths and code where answers can be checked programmatically.
- 17 Reasoning RL and R1-Style Training How DeepSeek-R1 and Kimi k1.5 demonstrated that pure reinforcement learning on verifiable rewards can elicit chain-of-thought reasoning in LLMs without any human-labelled reasoning traces.
- 18 Reinforcement Learning from Human Feedback How preference data and PPO turn a pretrained language model into a helpful, honest, harmless assistant.
- 19 Reward Model Calibration and Drift Reward models trained on human preferences suffer from miscalibration and distribution shift, causing the optimised policy to exploit proxy scores in ways that diverge from actual human intent.
- 20 The Infrastructure of LLM RL LLM post-training via RL requires four coordinated systems running simultaneously - a policy, a reference model, a reward model, and a value function - and the design choices for each determine both what behaviours emerge and where the training breaks.
- 21 The KL-Regularised RL Objective The KL-regularised RL objective balances reward maximisation against a penalty that keeps the policy close to a reference model, preventing reward hacking while allowing genuine improvement.