Offline & Model-Based RL
Learning from logged data, distribution shift, conservative value estimation, world models and planning.
5concepts
58flashcards
36minutes of reading
- 01 Conservative Value Estimation How pessimism turns an unverifiable estimate into a safe one, the mechanism behind conservative Q-learning, and why a lower bound is the right object when you cannot test.
- 02 Distribution Shift in Offline RL Why learning a policy from a fixed dataset fails in a way supervised learning does not, how value overestimation compounds through bootstrapping, and what makes this the central problem of the field.
- 03 Evaluating an Offline RL Policy Why the setting that forbids environment interaction also forbids the obvious way to compare policies, the estimators available, and why their variance is worst exactly where the decision matters.
- 04 Model-Based RL and Compounding Error Why learning a dynamics model buys sample efficiency, how one-step errors compound over a rollout, and the design choices that keep imagination useful.
- 05 Planning with a Learned Model How to use a dynamics model at decision time rather than for training, why MPC re-plans every step, and what tree search adds when the model is exact and when it is not.