RT-2 and Web-Scale Robot Learning

A robot has never in its life picked up a dinosaur toy, yet asked to "move the extinct animal to the group of animals" it does exactly that. It has never been trained to identify an improvised hammer, yet handed a scattering of objects it reaches for the rock. RT-2 (Brohan et al., 2023) is the concrete demonstration that these behaviours do not require robot demonstrations of each concept at all. They fall out of one idea: take a model that already learned about extinction and tools and rocks from the internet, and teach it to emit robot actions in the same breath as it emits words.

Actions as just more language

The mechanism is almost aggressively simple, and that is the point. RT-2 starts from a large vision-language model (VLM) already pretrained on internet-scale image-text data: visual question answering, captioning, the usual web corpus of pictures paired with language. Such a model maps an image and a text prompt to a text response. RT-2's move is to make a robot action be a text response.

A low-level action for the robots in the paper is a small vector: end-effector position and rotation deltas, gripper open/close, and a termination flag. RT-2 discretises each dimension into one of 256 bins and represents the whole action as a short string of tokens, so a command becomes a sequence like 1 128 91 241 5 101 127. The model predicts these tokens exactly as it would predict the tokens of a sentence. Nothing in the architecture distinguishes "answer the question" from "act"; both are next-token prediction over the same vocabulary.

That single representational choice is what makes the rest work. Because actions are tokens, the robot-control objective and the web vision-language objective are the same objective, so a single model can be trained on both at once.

Co-training, not fine-tuning-then-forgetting

The naive way to build a robot policy from a VLM is to fine-tune it on robot data until it stops doing anything else. RT-2 instead co-fine-tunes: it trains on robot trajectories and the original internet vision-language tasks together, in the same mixture. This is the load-bearing design decision.

The reason is catastrophic forgetting. Fine-tune a model exclusively on a few hundred thousand robot episodes and it becomes a competent but narrow policy that has quietly discarded most of what the web taught it; its semantic knowledge decays because nothing in the training signal rewards keeping it. By keeping web data in the mix, RT-2 preserves the VLM's grip on language and visual semantics while it learns control. The web knowledge and the robot skill share one set of weights and one training loss, and neither is allowed to erase the other. The generalisation the paper reports is a direct consequence: the model can act on concepts it only ever saw on the internet, because those concepts were never trained out of it.

What emerged

The paper frames RT-2's headline results as emergent capabilities, meaning skills present in the web data but absent from the robot demonstrations. Evaluated against a robot-data-only baseline, RT-2 showed roughly a 2x improvement in generalisation across several axes (novel objects, novel backgrounds, novel environments), and around 3x on the emergent-capability evaluations specifically. Three families of behaviour are worth naming:

• Novel objects. Instructions referring to objects never present in robot training, identified by the VLM's web-learned visual vocabulary.
• Novel instructions and semantic reasoning. Commands the robot was never shown, including ones requiring background knowledge (which object is the extinct animal, which drink for a tired person).
• Chain-of-thought control. When augmented with chain-of-thought reasoning, RT-2 can perform multi-stage semantic reasoning, first reasoning in language about what to do, then emitting the action. This lets it handle a command like "pick the object that could be used as a hammer" by reasoning to the rock and then acting.

Set this against RT-1 (Brohan et al., 2022), the immediate predecessor. RT-1 is an excellent robot-only transformer: trained on a large real-world demonstration dataset, it learns robust manipulation and studies how performance scales with data size and diversity. But its knowledge is bounded by what the robots did. It has no channel to the internet's semantics. RT-2's contribution is precisely that channel: same broad goal (a general, scalable robot policy), but the knowledge now flows in from web pretraining rather than only from teleoperation.

When it falls down

RT-2 generalises semantics, not motor skills, and conflating the two is the fastest way to over-read the demos.

• It cannot do what its body cannot do. RT-2 can newly recognise and reason about an object, but the physical action repertoire is still bounded by the robot skills present in the demonstration data. The paper is explicit that the model does not acquire new motor skills from web data; it applies its existing manipulation skills to novel semantic targets. Ask it for a manipulation the robot was never taught and semantics will not save you.
• A large model on a control loop is slow and costly. Running a multi-billion-parameter VLM to emit every action puts real inference latency and compute cost inside the control loop, a hard constraint for high-frequency or on-board control that a small policy network does not face.
• Closed data, limited reproducibility. The model builds on proprietary VLMs and a robot dataset that is not freely reproducible outside the lab that collected it. Independent replication and honest comparison are correspondingly hard, which is a standing caveat on any single lab's headline numbers.
• Curated demos are not field reliability. The reported behaviours are impressive on the evaluated task suites, but a controlled evaluation is cleaner than a kitchen. Success on generalisation benchmarks does not certify the per-attempt reliability an unattended deployment needs; treat the emergent-capability figures as evidence of a mechanism, not as a service-level guarantee.

Further reading

• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control - Brohan et al., 2023; the primary paper: action tokenisation, co-fine-tuning, and the generalisation and chain-of-thought results.
• RT-2 project page - the official DeepMind page with demonstration videos and the emergent-capability framing.
• RT-1: Robotics Transformer for Real-World Control at Scale - Brohan et al., 2022; the robot-only predecessor and the scaling-with-data study RT-2 builds on.