Vision-Language-Action Models

Ask a robot to "pick up the object that could work as a hammer" and, historically, it had no chance: nothing in its few thousand teleoperated demonstrations ever mentioned hammers, improvised tools, or the physics of striking. A vision-language-action (VLA) model can do it, not because it was trained on hammers, but because the vision-language backbone underneath it already read the internet. VLAs are the LLM playbook applied to control: take a model that learned the meaning of the visual world from web-scale image-text data, then teach it to emit actions instead of captions. The bet is that semantic generalisation transfers from language and vision into the body.

The recipe

A VLA is a vision-language model (VLM) with an action output bolted on and fine-tuned on robot demonstrations. The three moving parts:

1. Backbone. Start from a pretrained VLM: a vision encoder feeding a language model, already fluent in objects, spatial relations, and instructions. This is the source of everything the robot could not learn from its own data.
2. Action representation. Decide how the model expresses a motor command. Two families dominate, described below.
3. Fine-tuning. Train on teleoperated demonstrations: sequences of (image, instruction, action) tuples. The gradient teaches the VLM to ground its semantic knowledge in the specific robot's action space, ideally without overwriting the web knowledge that made it worth using.

The input at run time is a set of camera images plus a natural-language instruction. The output is an action, typically an end-effector pose delta or joint targets, produced at whatever frequency the control loop demands.

Actions as tokens, or actions as a continuous head

The central design question is how a language model, whose native output is a discrete token, produces a real-valued 7-DoF motor command. Two answers define the field.

Actions as text tokens. RT-2 (Google DeepMind, 2023) discretises each action dimension into bins and treats those bins as extra vocabulary tokens, so the action is literally emitted as text: the same next-token machinery that writes a caption now writes a trajectory. The elegance is that no architecture changes; robot trajectories are co-fine-tuned alongside ordinary vision-language tasks like visual question answering, which keeps the web knowledge alive. RT-2 reported roughly 2x the generalisation of its non-VLM predecessor on novel objects and could follow commands absent from its robot data. OpenVLA (2024, open-source) took the same tokenised-action approach to a 7B model trained on around 970k real demonstrations, and reported beating the much larger closed RT-2-X on task success while shipping the weights, code, and fine-tuning recipe publicly. (See the neighbouring concepts on RT-2 and on action tokenisation for the mechanics.)

Actions as a continuous head. Discretising throws away precision and caps how fast you can act, because you decode one token per action dimension. The generalist-policy line represented by Physical Intelligence's pi-0 (2024) instead augments the VLM with a continuous action output via flow matching, a diffusion variant, letting a 3B VLM backbone emit smooth motor commands at up to 50 Hz across multiple robot embodiments (see the imitation and diffusion policy concept for why iterative denoising suits multimodal action distributions). This is the push toward a single generalist policy: one model trained on data from many robots, aiming to be a "foundation model" for control rather than a per-task policy.

The split matters. Tokenised actions reuse the LLM stack unchanged and inherit its tooling; continuous heads buy precision and control frequency at the cost of a bespoke output module. Which wins depends on whether your bottleneck is semantic generalisation or dexterous, high-frequency manipulation.

Why borrow from a VLM at all

The honest justification is data scarcity. A frontier language model trains on trillions of tokens scraped for free; the largest open robot corpus is on the order of a million demonstrations, each collected by a human physically teleoperating a machine in real time. You cannot scrape grasping. So the strategy is to get the expensive knowledge (what a spatula is, that "spilled" means liquid on a surface, that a knife is grasped by the handle) from the domain where data is abundant, and spend the scarce robot data only on the residual: how this arm, with these joints, enacts what the model already understands. When it works, the robot generalises to objects, phrasings, and scenes it never saw in a single demonstration, because that generalisation was never learned from robot data in the first place.

When it falls down

• Robot data is scarce and expensive. There is no web-scale corpus of actions. Every demonstration is human teleoperation, slow and costly, so robot datasets are millions of samples where language models have trillions of tokens. Fine-tuning on a thin, narrow dataset is the binding constraint, and it is why so much of the field is really data-collection engineering.
• Real-time control fights model size. A big VLM is slow, and a manipulation loop may need tens of actions per second. Inference latency directly caps dexterity: a model that reasons beautifully at 2 Hz cannot catch a falling object. This tension drives the continuous-head, high-frequency designs and heavy inference optimisation, and it means the biggest, smartest backbone is often not deployable.
• The embodiment gap. A policy trained on one robot's morphology, its link lengths, gripper, camera placement, and action space, does not transfer cleanly to a different body. Cross-embodiment training (pooling data across robot types) helps but does not dissolve the problem; a demonstration on a Franka arm is not a demonstration for a mobile bimanual platform.
• Brittleness to physical distribution shift. Lighting, clutter, a novel object pose, or a slightly different table height can break a policy that scored well in its training distribution. The physical world has a longer tail than a benchmark, and unlike a chatbot's wrong answer, a wrong action can knock things over or damage hardware.
• Evaluation is hard and often not reproducible. Real-robot evaluation depends on the exact hardware, scene setup, and object set, so numbers rarely transfer between labs and are laborious to reproduce. Simulation is cheaper but opens the sim-to-real gap (its own concept). Reported success rates should be read as "on this robot, in this lab, on these tasks", not as portable scores.

Further reading

• RT-2: Vision-Language-Action Models - Google DeepMind's project page for the actions-as-text-tokens approach and the co-fine-tuning recipe that preserves web knowledge.
• OpenVLA: An Open-Source Vision-Language-Action Model - a 7B open VLA trained on ~970k demonstrations, with public weights and an efficient fine-tuning recipe.
• pi-0: A Vision-Language-Action Flow Model for General Robot Control - Physical Intelligence's generalist policy using flow-matching continuous actions at up to 50 Hz across multiple robot embodiments.