RL for Language Models
RLHF as an RL problem, KL-regularised objectives, GRPO, RLVR, and reward over-optimisation.
21concepts
142flashcards
172minutes of reading
- 01 Best-of-N and Inference-Time Selection Best-of-N sampling generates multiple completions from a language model and returns the one ranked highest by a reward model, trading inference compute for quality without updating any weights.
- 02 Length and Format Reward Hacking When a reward model assigns higher scores to longer or more structured responses regardless of quality, RL training exploits that signal and the policy degrades into verbose padding and hollow formatting instead of improving reasoning.
- 03 Process vs Outcome Rewards Process reward models score each reasoning step individually, giving denser training signal than outcome rewards and catching errors before they contaminate a final answer.
- 04 RLHF as a Reinforcement-Learning Problem RLHF recasts language model alignment as a policy-optimisation problem where a reward model trained on human comparisons provides the scalar signal that PPO uses to update the language policy.
- 05 Rejection Sampling as RL Rejection sampling fine-tuning filters a model's own outputs by correctness and trains on the survivors, achieving a policy-improvement step that is mathematically equivalent to one round of RL but without an explicit optimiser loop.
- 06 Reward Models as Learned Rewards A reward model is a classifier trained on human preference comparisons that outputs a scalar score, standing in for the true human utility function during RL fine-tuning of language models.
- 07 Reward Over-Optimisation When a language model is trained too aggressively against a proxy reward model, it learns to exploit the proxy rather than genuinely improve, causing measured reward to climb while true quality declines.
- 08 The Bandit Framing of RLHF RLHF treats a language model as a contextual bandit that receives a single scalar reward per complete response, making full Markov decision process machinery unnecessary but also hiding the dangers of reward over-optimisation.