Planning with a Learned Model

A learned dynamics model can be used two ways: to generate synthetic experience for training a policy, or to search over action sequences at decision time. The second is planning, and it has a property the first lacks: it can respond to the current situation with computation rather than relying on what a policy network memorised.

Model predictive control

The pattern is simple and effective. At each step, sample or optimise candidate action sequences over a short horizon, evaluate each by rolling it out through the model, execute only the first action of the best sequence, and re-plan at the next step from the actual observed state.

Executing one step and re-planning is what makes it robust. The plan is only trusted for one step, so model error has one step to accumulate before being corrected by a real observation. The horizon bounds how far compounding error can go, and the re-planning corrects drift continuously.

The cross-entropy method is the usual optimiser: sample sequences from a distribution, keep the best fraction, refit the distribution to them, repeat. It is derivative-free, parallelises across samples, and handles the non-differentiable and multimodal objectives that arise.

The cost is computation at every decision, which scales with samples times horizon times model evaluation cost. For a real-time control loop this is the binding constraint, and it is why planning appears in robotics at modest control rates and rarely in high-frequency settings.

Tree search

Where actions are discrete and the model is exact, Monte Carlo tree search is far more effective than sampling flat sequences, because it concentrates computation on promising branches and reuses computation across the tree.

MuZero extended this to learned models with an important twist: rather than modelling the environment's state, it learns a latent representation trained only to be useful for predicting reward, value and policy. The model need not reconstruct observations at all, which removes the requirement to model perceptual detail irrelevant to the decision.

Combining planning and learning

Planning at decision time and a learned value function are complementary. The value function provides a leaf estimate so the search does not need to reach a terminal state, and the search improves on the policy, producing targets that train the policy to imitate the search. This loop, search improves policy, policy guides search, is the structure behind the strongest game-playing systems.

When it breaks

Long horizons are computationally and statistically infeasible. Sampling the space of long action sequences is exponentially hard and model error accumulates, so planning horizons are short and a learned value function is what supplies the long-term component.

Continuous high-dimensional actions are hard to search. Sampling covers such spaces poorly, so the optimiser matters more than the model, and gradient-based planning requires a differentiable model with its own conditioning problems.

Planning cost is paid per decision. Unlike training-time model use, it appears in the deployed latency budget, which frequently rules it out for interactive systems regardless of its accuracy advantage.

Real-time constraints force truncation. A planner with a compute budget produces the best plan it found rather than the best plan, so the quality is a function of available time, and behaviour degrades under load in a way a reactive policy does not.