Flow-Matching Action Experts

RT-2 turned robot actions into text by discretising each dimension into 256 bins and emitting them as tokens. That choice bought enormous leverage, because it made a robot policy just another vision-language model, but it also imposed a hard ceiling. Every action step costs a full autoregressive decode of several tokens, and folding laundry needs an action every 20 milliseconds. Per-dimension, per-timestep binning also degrades badly on high-frequency dexterous data, where consecutive actions are highly correlated and the binning throws that structure away (Pertsch et al., 2025, arXiv:2501.09747).

Two responses to this exist, and they are worth understanding as a pair: compress the tokens, or stop tokenising.

Stop tokenising: the action expert

π₀ keeps a pretrained vision-language model as the perception and language backbone and bolts on a separate set of weights, an action expert, that emits continuous actions directly by flow matching. The published configuration is a 3B PaliGemma backbone plus a 300M action expert, 3.3B parameters in total, producing an action chunk of horizon \(H = 50\) at up to 50 Hz for dexterous tasks such as laundry folding, trained on a mixture including over 10,000 hours of robot data across 7 robot configurations spanning single-arm, dual-arm and mobile platforms (Black et al., 2024, arXiv:2410.24164).

The action head is trained with conditional flow matching. During training the network regresses a denoising vector field over noised action chunks conditioned on the observation and instruction; at inference the chunk is produced by integrating that field from \(\tau = 0\) to \(\tau = 1\) starting from noise, using forward Euler with 10 steps. Ten network evaluations per chunk of 50 actions is the arithmetic that makes 50 Hz achievable: the expensive part runs once per chunk, not once per action.

Why a generative head rather than regression? Because the correct action is multimodal. Two demonstrators shown the same cluttered scene will grasp from opposite sides, and a mean-squared-error policy averages those into a motion that hits the object dead centre and fails. Modelling a distribution over chunks is the same argument that made diffusion policies work, at an average improvement of 46.9% across 12 tasks from 4 benchmarks (Chi et al., RSS 2023, arXiv:2303.04137).

Compress the tokens: FAST

The alternative keeps the autoregressive interface and fixes the tokeniser. FAST applies a discrete cosine transform to the action chunk, quantises the coefficients, and byte-pair-encodes the result, so a smooth 50-step trajectory becomes a short token string instead of 350 near-duplicate bin indices. It scales to training on 10,000 hours of robot data, matches diffusion VLA performance, and reduces training time by up to 5×; the released FAST+ tokeniser was fit on 1M real trajectories and is meant to be used off the shelf.

The trade is straightforward. The action expert gives the lowest inference latency per chunk and the cleanest continuous output; FAST keeps the pure language-model stack, which means every existing VLM training and serving tool works unmodified.

Where it breaks

Chunking trades reactivity for smoothness. Committing to 50 actions means the robot is open-loop for the duration of the chunk. Execute the whole chunk and it cannot react to a slipping object; re-plan every step and you pay 10 network evaluations per control step. Real systems execute a prefix of the chunk and re-plan, and that prefix length is a genuine latency-versus-reactivity dial, the same receding-horizon idea that Diffusion Policy uses.

Solver steps are a quality knob nobody reports consistently. Ten Euler steps is a choice. Fewer steps degrade chunk quality in ways that look like clumsiness rather than error, and comparisons between policies at different step counts are not comparisons of policies.

The backbone can be starved. Adding a separate action expert means the language backbone's gradients come partly from action loss. Too much action data and the model's semantic generalisation, the entire reason a VLM was used, erodes. Mixture ratios matter as much here as in cross-embodiment training.

Continuous output hides failure. A discrete-token policy can express uncertainty through its token distribution. A flow head always returns a confident-looking trajectory, so detecting "the policy does not know what to do here" requires separate machinery such as ensemble disagreement or chunk-to-chunk variance.