Model-Based RL and Compounding Error
Why learning a dynamics model buys sample efficiency, how one-step errors compound over a rollout, and the design choices that keep imagination useful.
Model-free RL learns a value function or policy directly from experience and needs a great deal of it. Model-based RL learns the environment's dynamics, then plans or trains inside the learned model, generating as much synthetic experience as compute allows. The sample efficiency gain is often an order of magnitude or more, and the price is a specific and unavoidable failure mode.
The compounding problem
A dynamics model predicts the next state from the current state and action. Any error in that prediction moves the imagined trajectory slightly off the real one, and the next prediction is made from an already-wrong state, which is out of the model's training distribution and therefore predicted worse.
Errors compound superlinearly with rollout length. A model with small one-step error can produce trajectories that are physically implausible after fifty steps, and a policy optimised against those trajectories exploits the model's errors rather than solving the task. The classic symptom is a policy that achieves enormous imagined return and fails immediately in the real environment.
What controls it
Short rollouts. Branching short model rollouts from states in the real replay buffer, rather than generating long trajectories from an initial state, keeps every imagined step close to real data. This is the core idea of model-based policy optimisation and it is the single most effective mitigation.
Ensembles. Training several dynamics models and using their disagreement as an uncertainty estimate lets the algorithm penalise or truncate rollouts entering regions where the models disagree, which is exactly where they are unreliable.
Latent-space models. Predicting forward in a learned latent space rather than in observation space avoids modelling perceptual detail irrelevant to control, which is what makes the approach work on pixel observations.
Uncertainty-penalised reward. In the offline setting, penalising imagined reward by model uncertainty implements pessimism through the dynamics model, which is how offline model-based methods avoid the same distribution shift problem that afflicts model-free ones.
When it breaks
Policy optimisation seeks model errors. An optimiser given a differentiable imagined environment will find its exploitable flaws, which is adversarial pressure on exactly the component that is approximate. This is why model exploitation is a named failure rather than an occasional accident.
Aleatoric and epistemic uncertainty need separating. Stochastic dynamics are irreducibly uncertain and should not cause a rollout to be truncated; model ignorance should. Ensembles conflate the two unless the individual models are probabilistic, in which case the spread of means measures epistemic and the average variance measures aleatoric.
Sample efficiency is bought with compute. Model-based methods use far fewer environment interactions and far more computation, so they win where interaction is expensive, robots, medicine, industrial systems, and lose where a simulator is cheap and fast.
Model accuracy is not the objective. A model can predict states well while being useless for control if it is inaccurate in the dimensions that determine reward. Evaluating a dynamics model by prediction error rather than by the performance of the policy it produces is a common and misleading choice.
12 flashcards for this concept
Click a card to reveal the answer.