RL Foundations
MDPs, value functions, TD learning, policy gradients, actor-critic, TRPO and PPO.
20concepts
140flashcards
146minutes of reading
- 01 Generalised Advantage Estimation GAE introduces a single hyperparameter lambda that smoothly interpolates between high-bias/low-variance TD(0) and low-bias/high-variance Monte Carlo advantage estimates, making policy gradient training substantially more stable.
- 02 Model-Based Reinforcement Learning Model-based RL learns an explicit dynamics model of the environment and uses it for planning or synthetic data generation, trading model bias for dramatic gains in sample efficiency.
- 03 Proximal Policy Optimisation PPO stabilises policy gradient training by clipping the probability ratio between old and new policies, preventing destructively large updates without the computational overhead of second-order methods.
- 04 Trust-Region Policy Optimisation TRPO is a policy-gradient algorithm that enforces a KL-divergence constraint on each update, guaranteeing monotonic policy improvement and preventing the catastrophic performance collapses that plague vanilla gradient ascent.