Imitation Learning and Diffusion Policies

Teach a robot by showing it. Collect a few hundred demonstrations of a human driving the arm through a task, train a network to map each observed state to the action the human took, and you have a policy. This is behaviour cloning, and it is the most natural idea in robot learning. It is also the idea with the sharpest hidden failure: a policy trained this way does not fail gracefully as it gets slightly worse, it fails catastrophically, because a small error moves the robot into a state the human never visited, and there the policy has no idea what to do. Ross, Gordon and Bagnell made this precise in 2010, and the fix is not more data in the naive sense; it is a different way of modelling what a demonstration even is.

Behaviour cloning and why the errors compound

Behaviour cloning treats control as plain supervised learning. Given demonstration pairs (state, action), fit a function pi(a | s) that reproduces the expert's action. Training is standard: minimise the loss on the demonstration set. The trouble is that supervised learning assumes the test inputs are drawn from the same distribution as the training inputs, and in sequential control that assumption is false the moment the policy starts acting.

Here is the mechanism. Suppose the trained policy makes a mistake with small probability at each step. A mistake nudges the robot slightly off the demonstrated trajectory, into a state that is a little unfamiliar. Because that state is under-represented in the training data, the policy is more likely to err there, which pushes it further off-distribution, which makes the next error still more likely. The distribution of states the policy actually visits drifts away from the distribution it was trained on. This is covariate shift, and its consequence is quantitative.

Ross et al. proved that if a behaviour-cloned policy has per-step error rate epsilon over a task of horizon T, the total cost can grow as O(epsilon * T^2), not the O(epsilon * T) you would hope for. The extra factor of T is the compounding: an early mistake is not a local blip, it changes every state the policy sees for the rest of the episode. Doubling the task length can quadruple the failure, which is why a policy that looks flawless on short clips falls apart on a long horizon.

Their algorithm, DAgger (Dataset Aggregation), attacks the shift directly. Run the current policy, let it visit its own (imperfect) states, ask the expert what the correct action would have been in each of those states, add those labels to the dataset, and retrain. Iterate. By labelling the states the policy actually reaches rather than only the states the expert reached, DAgger trains on its own error distribution and collapses the quadratic term back toward linear. The catch is operational: DAgger needs an expert available online to label novel states, which is cheap in a simulator and painful with a human teleoperating a real arm.

Multimodality: why averaging destroys the policy

DAgger fixes coverage. It does not fix a second, subtler problem with how behaviour cloning represents the action itself. Human demonstrations are multimodal: there are often several equally good ways to do a thing. Reaching around a mug, you might go left or right; both are correct, neither is a compromise between them. If your policy is a standard regressor trained with a mean-squared-error loss, it learns the conditional mean of the demonstrated actions. The mean of "go left" and "go right" is "go straight into the mug". A unimodal model asked to fit a multimodal target averages the modes into an action that satisfies none of them.

This is not a corner case; it is pervasive in manipulation, where tasks routinely admit multiple valid strategies. The fix is to stop predicting a single action and start modelling the full distribution of good actions, so the policy can commit to one mode instead of averaging across them. That is precisely what a generative policy does.

Diffusion policies and action chunking

A diffusion policy, introduced by Chi et al. in 2023, represents the robot's action-generation as a conditional denoising diffusion process. Instead of regressing an action directly, it starts from Gaussian noise and iteratively denoises it into an action, conditioned on the current observation. Because diffusion models are built to capture complex, multimodal distributions, the policy can represent "go left" and "go right" as two distinct modes and sample one cleanly, rather than collapsing to the mean. Across twelve tasks from four manipulation benchmarks the authors reported an average improvement of roughly 47 percent over prior methods, with the multimodality handling doing much of the work.

Two design choices make it practical. First, the policy denoises a short sequence of future actions, not a single step, and executes them with receding-horizon control (predict a horizon, execute the first part, re-plan). Second, it conditions the denoising on observations rather than mixing observations into the diffusion, which keeps inference tractable.

Predicting a chunk of actions is a general trick, and it is the core of ACT (Action Chunking with Transformers) from Zhao et al., 2023, the method behind the low-cost ALOHA bimanual platform. ACT learns a generative model over short action sequences and executes them as a unit. The point is directly about compounding error: if the policy commits to a chunk of k actions per decision, it makes a fresh, error-prone decision T/k times instead of T times, so the number of opportunities to drift off-distribution drops by a factor of k. Chunking trades a little reactivity for a large reduction in the compounding term. On the ALOHA hardware, ACT reached 80 to 90 percent success on fine bimanual tasks from around ten minutes of demonstrations, a regime where a naive step-wise clone would drift badly.

The two ideas are complementary. Generative modelling (diffusion, or the CVAE inside ACT) fixes the multimodality problem so the policy commits to a coherent strategy; chunking shortens the decision horizon so drift has fewer chances to accumulate. Neither requires an online expert the way DAgger does, which is why they became the default recipe for learning from a fixed set of teleoperated demonstrations.

When it falls down

• Demonstration coverage is still the ceiling. Neither a diffusion policy nor action chunking invents behaviour for a state with no nearby demonstration. Generative modelling fixes how the action is represented, not the absence of data. Drive the robot far enough off the demonstrated manifold and it has nothing to interpolate from.
• Sampling adds latency. A diffusion policy denoises over multiple steps to produce each action sequence, so inference is heavier than a single forward pass. On a control loop that wants tens of hertz this matters, and it is why fewer-step samplers and distillation are active work.
• Chunking trades away reactivity. Committing to k actions means the policy cannot respond to a disturbance that arrives mid-chunk until the chunk ends. Longer chunks cut compounding error but slow reaction to surprises; the horizon is a tuning knob, not a free win.
• Reset-to-recover is assumed away. The compounding-error analysis and most manipulation results assume episodes that begin near demonstrated start states. A policy that wanders far off-distribution rarely recovers on its own, and evaluation that always resets to clean initial conditions hides this.
• Cloning cannot beat the demonstrator. Imitation learning targets the expert's behaviour; its best case is to match the human, not to exceed them, and it inherits the human's mistakes and inefficiencies. Surpassing the demonstrator requires a reward signal and reinforcement learning on top, at which point you leave pure imitation behind.

Further reading

• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning - Ross, Gordon and Bagnell; the covariate-shift analysis, the compounding-error bound, and the DAgger algorithm.
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion - Chi et al.; representing the policy as conditional action denoising to capture multimodal action distributions.
• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware - Zhao et al.; the ACT action-chunking method and the ALOHA platform, learning fine bimanual tasks from a handful of demonstrations.