Concept library
446 concepts across 8 domains and 36 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
04
Reinforcement Learning
Classical RL, then the specific dialect of it that post-trains language models.
2tracks
41concepts
282cards
5.3hreading
RL Foundations MDPs, value functions, TD learning, policy gradients, actor-critic, TRPO and PPO. 20 concepts · 140 cards
- 01 Returns, Discounting, and Episodes The return is the quantity a reinforcement learning agent actually optimises; discounting controls how far into the future it looks, and whether interactions are episodic or continuing shapes which formulation applies.
- 02 Actor-Critic Methods Actor-critic methods combine a policy network (actor) with a value estimator (critic) to reduce variance in policy gradient updates without the high bias of pure value-based methods.
- 03 Baselines and Variance Reduction A baseline is a state-dependent function subtracted from the return in policy gradient updates to reduce estimator variance without introducing bias.
- 04 Deep Q-Networks DQN combines Q-learning with a deep convolutional network and two stabilisation tricks (experience replay and a target network) to learn Atari-level control policies directly from raw pixels.
- 05 Dynamic Programming for RL Dynamic programming solves RL problems exactly by bootstrapping value estimates across states using the Bellman equations, but only when you have a perfect model of the environment.
- 06 Entropy Regularisation Entropy regularisation adds a bonus term to the RL objective that rewards stochastic policies, improving exploration and preventing premature convergence to deterministic optima.
- 07 Markov Decision Processes A Markov Decision Process is the formal framework that turns vague "learn from interaction" intuitions into a precise mathematical problem a computer can solve.
- 08 Monte Carlo Methods Monte Carlo methods estimate value functions by averaging complete episode returns, making them the simplest model-free approach but one that requires episodic tasks and carries high variance.
- 09 On-Policy vs Off-Policy Learning On-policy methods learn from data collected by the policy being updated, while off-policy methods learn from data generated by a different behaviour policy, enabling experience reuse but requiring bias corrections.
- 10 Policy Gradients and REINFORCE Policy gradient methods directly optimise a stochastic policy by estimating the gradient of expected return through sampled trajectories, sidestepping the need to represent a value function over every state-action pair.
- 11 Policy Methods vs Value Methods Policy methods optimise a parameterised policy directly via gradient ascent on expected return, while value methods learn a value function and derive behaviour from it; the distinction shapes sample efficiency, stability, and action-space suitability across the entire RL algorithm landscape.
- 12 Q-Learning Q-learning is a model-free, off-policy temporal-difference algorithm that estimates the value of (state, action) pairs and converges to an optimal policy without requiring a model of the environment.
- 13 Reward Shaping and Credit Assignment Reward shaping injects domain knowledge into the reward signal to speed up learning, while credit assignment determines which past actions actually caused a delayed reward.
- 14 Temporal-Difference Learning TD learning combines the trial-and-error sampling of Monte Carlo with the bootstrapped updates of dynamic programming to learn value functions online, without waiting for episode ends.
- 15 The Exploration-Exploitation Trade-off Choosing when to try something new versus repeating what already works is the central tension in reinforcement learning, and getting it wrong kills agent performance regardless of how well the rest of the system is designed.
- 16 Value Functions and the Bellman Equations Value functions assign expected cumulative reward to states and state-action pairs; the Bellman equations express these values as self-consistent recursive relationships that underpin every practical RL algorithm.
- 17 Generalised Advantage Estimation GAE introduces a single hyperparameter lambda that smoothly interpolates between high-bias/low-variance TD(0) and low-bias/high-variance Monte Carlo advantage estimates, making policy gradient training substantially more stable.
- 18 Model-Based Reinforcement Learning Model-based RL learns an explicit dynamics model of the environment and uses it for planning or synthetic data generation, trading model bias for dramatic gains in sample efficiency.
- 19 Proximal Policy Optimisation PPO stabilises policy gradient training by clipping the probability ratio between old and new policies, preventing destructively large updates without the computational overhead of second-order methods.
- 20 Trust-Region Policy Optimisation TRPO is a policy-gradient algorithm that enforces a KL-divergence constraint on each update, guaranteeing monotonic policy improvement and preventing the catastrophic performance collapses that plague vanilla gradient ascent.
RL for Language Models RLHF as an RL problem, KL-regularised objectives, GRPO, RLVR, and reward over-optimisation. 21 concepts · 142 cards
- 01 Best-of-N and Inference-Time Selection Best-of-N sampling generates multiple completions from a language model and returns the one ranked highest by a reward model, trading inference compute for quality without updating any weights.
- 02 Length and Format Reward Hacking When a reward model assigns higher scores to longer or more structured responses regardless of quality, RL training exploits that signal and the policy degrades into verbose padding and hollow formatting instead of improving reasoning.
- 03 Process vs Outcome Rewards Process reward models score each reasoning step individually, giving denser training signal than outcome rewards and catching errors before they contaminate a final answer.
- 04 RLHF as a Reinforcement-Learning Problem RLHF recasts language model alignment as a policy-optimisation problem where a reward model trained on human comparisons provides the scalar signal that PPO uses to update the language policy.
- 05 Rejection Sampling as RL Rejection sampling fine-tuning filters a model's own outputs by correctness and trains on the survivors, achieving a policy-improvement step that is mathematically equivalent to one round of RL but without an explicit optimiser loop.
- 06 Reward Models as Learned Rewards A reward model is a classifier trained on human preference comparisons that outputs a scalar score, standing in for the true human utility function during RL fine-tuning of language models.
- 07 Reward Over-Optimisation When a language model is trained too aggressively against a proxy reward model, it learns to exploit the proxy rather than genuinely improve, causing measured reward to climb while true quality declines.
- 08 The Bandit Framing of RLHF RLHF treats a language model as a contextual bandit that receives a single scalar reward per complete response, making full Markov decision process machinery unnecessary but also hiding the dangers of reward over-optimisation.
- 09 Credit Assignment over Long Generations Explains why distributing a single scalar reward back across hundreds of generation steps is the central unsolved tension in RL for language models, and surveys the main strategies used to address it.
- 10 Evaluating RL-Tuned Models Standard NLP benchmarks break silently when applied to RL-tuned models because the training objective optimises for a reward signal that can diverge from genuine capability, requiring a distinct evaluation stack to distinguish real improvement from reward gaming.
- 11 Exploration in Language-Model RL Language-model RL training collapses silently when the policy stops generating diverse completions, and standard RL exploration techniques must be reinterpreted to work inside a token-sequence action space.
- 12 GRPO: Group Relative Policy Optimisation GRPO removes the critic network from PPO by estimating baselines from a sampled group of outputs, halving the GPU footprint while delivering competitive reasoning improvements.
- 13 Multi-Turn and Agentic RL Multi-turn and agentic RL extends single-response RLHF to sequences of actions across environment steps, requiring credit assignment, trajectory-level rewards, and new training algorithms suited to long-horizon tool-using agents.
- 14 Offline RL and the DPO Connection DPO re-derives the standard KL-regularised RLHF objective and solves it in closed form, turning preference alignment into a supervised classification loss over offline data without ever sampling from the policy during training.
- 15 PPO for Language Models Proximal Policy Optimisation clips the policy update ratio to prevent destructive gradient steps, making it the workhorse algorithm for RLHF fine-tuning of large language models.
- 16 RLVR: RL from Verifiable Rewards RLVR replaces the trained reward model in RLHF with an automated verifier that checks correctness against a ground-truth answer, producing a clean binary signal that sidesteps reward hacking and scales to tasks like maths and code where answers can be checked programmatically.
- 17 Reasoning RL and R1-Style Training How DeepSeek-R1 and Kimi k1.5 demonstrated that pure reinforcement learning on verifiable rewards can elicit chain-of-thought reasoning in LLMs without any human-labelled reasoning traces.
- 18 Reinforcement Learning from Human Feedback How preference data and PPO turn a pretrained language model into a helpful, honest, harmless assistant.
- 19 Reward Model Calibration and Drift Reward models trained on human preferences suffer from miscalibration and distribution shift, causing the optimised policy to exploit proxy scores in ways that diverge from actual human intent.
- 20 The Infrastructure of LLM RL LLM post-training via RL requires four coordinated systems running simultaneously - a policy, a reference model, a reward model, and a value function - and the design choices for each determine both what behaviours emerge and where the training breaks.
- 21 The KL-Regularised RL Objective The KL-regularised RL objective balances reward maximisation against a penalty that keeps the policy close to a reference model, preventing reward hacking while allowing genuine improvement.