RL Foundations
MDPs, value functions, TD learning, policy gradients, actor-critic, TRPO and PPO.
20concepts
140flashcards
146minutes of reading
- 01 Actor-Critic Methods Actor-critic methods combine a policy network (actor) with a value estimator (critic) to reduce variance in policy gradient updates without the high bias of pure value-based methods.
- 02 Baselines and Variance Reduction A baseline is a state-dependent function subtracted from the return in policy gradient updates to reduce estimator variance without introducing bias.
- 03 Deep Q-Networks DQN combines Q-learning with a deep convolutional network and two stabilisation tricks (experience replay and a target network) to learn Atari-level control policies directly from raw pixels.
- 04 Dynamic Programming for RL Dynamic programming solves RL problems exactly by bootstrapping value estimates across states using the Bellman equations, but only when you have a perfect model of the environment.
- 05 Entropy Regularisation Entropy regularisation adds a bonus term to the RL objective that rewards stochastic policies, improving exploration and preventing premature convergence to deterministic optima.
- 06 Markov Decision Processes A Markov Decision Process is the formal framework that turns vague "learn from interaction" intuitions into a precise mathematical problem a computer can solve.
- 07 Monte Carlo Methods Monte Carlo methods estimate value functions by averaging complete episode returns, making them the simplest model-free approach but one that requires episodic tasks and carries high variance.
- 08 On-Policy vs Off-Policy Learning On-policy methods learn from data collected by the policy being updated, while off-policy methods learn from data generated by a different behaviour policy, enabling experience reuse but requiring bias corrections.
- 09 Policy Gradients and REINFORCE Policy gradient methods directly optimise a stochastic policy by estimating the gradient of expected return through sampled trajectories, sidestepping the need to represent a value function over every state-action pair.
- 10 Policy Methods vs Value Methods Policy methods optimise a parameterised policy directly via gradient ascent on expected return, while value methods learn a value function and derive behaviour from it; the distinction shapes sample efficiency, stability, and action-space suitability across the entire RL algorithm landscape.
- 11 Q-Learning Q-learning is a model-free, off-policy temporal-difference algorithm that estimates the value of (state, action) pairs and converges to an optimal policy without requiring a model of the environment.
- 12 Reward Shaping and Credit Assignment Reward shaping injects domain knowledge into the reward signal to speed up learning, while credit assignment determines which past actions actually caused a delayed reward.
- 13 Temporal-Difference Learning TD learning combines the trial-and-error sampling of Monte Carlo with the bootstrapped updates of dynamic programming to learn value functions online, without waiting for episode ends.
- 14 The Exploration-Exploitation Trade-off Choosing when to try something new versus repeating what already works is the central tension in reinforcement learning, and getting it wrong kills agent performance regardless of how well the rest of the system is designed.
- 15 Value Functions and the Bellman Equations Value functions assign expected cumulative reward to states and state-action pairs; the Bellman equations express these values as self-consistent recursive relationships that underpin every practical RL algorithm.