RL for Language Models
RLHF as an RL problem, KL-regularised objectives, GRPO, RLVR, and reward over-optimisation.
21concepts
142flashcards
172minutes of reading
- 01 Credit Assignment over Long Generations Explains why distributing a single scalar reward back across hundreds of generation steps is the central unsolved tension in RL for language models, and surveys the main strategies used to address it.
- 02 Evaluating RL-Tuned Models Standard NLP benchmarks break silently when applied to RL-tuned models because the training objective optimises for a reward signal that can diverge from genuine capability, requiring a distinct evaluation stack to distinguish real improvement from reward gaming.
- 03 Exploration in Language-Model RL Language-model RL training collapses silently when the policy stops generating diverse completions, and standard RL exploration techniques must be reinterpreted to work inside a token-sequence action space.
- 04 GRPO: Group Relative Policy Optimisation GRPO removes the critic network from PPO by estimating baselines from a sampled group of outputs, halving the GPU footprint while delivering competitive reasoning improvements.
- 05 Multi-Turn and Agentic RL Multi-turn and agentic RL extends single-response RLHF to sequences of actions across environment steps, requiring credit assignment, trajectory-level rewards, and new training algorithms suited to long-horizon tool-using agents.
- 06 Offline RL and the DPO Connection DPO re-derives the standard KL-regularised RLHF objective and solves it in closed form, turning preference alignment into a supervised classification loss over offline data without ever sampling from the policy during training.
- 07 PPO for Language Models Proximal Policy Optimisation clips the policy update ratio to prevent destructive gradient steps, making it the workhorse algorithm for RLHF fine-tuning of large language models.
- 08 RLVR: RL from Verifiable Rewards RLVR replaces the trained reward model in RLHF with an automated verifier that checks correctness against a ground-truth answer, producing a clean binary signal that sidesteps reward hacking and scales to tasks like maths and code where answers can be checked programmatically.
- 09 Reasoning RL and R1-Style Training How DeepSeek-R1 and Kimi k1.5 demonstrated that pure reinforcement learning on verifiable rewards can elicit chain-of-thought reasoning in LLMs without any human-labelled reasoning traces.
- 10 Reinforcement Learning from Human Feedback How preference data and PPO turn a pretrained language model into a helpful, honest, harmless assistant.
- 11 Reward Model Calibration and Drift Reward models trained on human preferences suffer from miscalibration and distribution shift, causing the optimised policy to exploit proxy scores in ways that diverge from actual human intent.
- 12 The Infrastructure of LLM RL LLM post-training via RL requires four coordinated systems running simultaneously - a policy, a reference model, a reward model, and a value function - and the design choices for each determine both what behaviours emerge and where the training breaks.
- 13 The KL-Regularised RL Objective The KL-regularised RL objective balances reward maximisation against a penalty that keeps the policy close to a reference model, preventing reward hacking while allowing genuine improvement.