Cross-Embodiment Transfer

Language models got the internet. Vision models got LAION. Robotics got nothing comparable, because every dataset was collected on one robot, in one lab, with one gripper, one camera placement and one control convention, and was therefore useless to everyone else. The field spent a decade with hundreds of small, mutually incompatible datasets and no way to add them up.

Open X-Embodiment is the attempt to add them up: data pooled from 22 robot embodiments across 21 institutions, covering 527 skills over 160,266 tasks, with RT-X models trained on the union (Open X-Embodiment Collaboration, 2023, arXiv:2310.08864). The headline result is that a policy trained on the mixture outperforms the same policy trained only on the target robot's own data. Experience on other people's robots transfers.

What makes transfer possible at all

Two robots share almost nothing at the signal level. One publishes joint velocities at 100 Hz, another Cartesian deltas at 10 Hz; grippers range from parallel jaws to five-fingered hands; cameras sit on the wrist, over the shoulder, or both. Making a single policy consume all of this requires forcing agreement on three things.

A common action space. The usual choice is end-effector delta pose plus a gripper scalar, normalised per dataset, typically by quantiles rather than min-max so that a handful of outlier trajectories do not compress everything else into a narrow band. Joint-space data is converted through the robot's kinematics where possible. What survives this projection is coarse motion; what is lost is anything embodiment-specific, which is why fine dexterity transfers poorly.

A common observation interface. Images plus a language instruction. Proprioception, force sensing and depth are frequently dropped precisely because not every dataset has them, which is a real information sacrifice made for the sake of poolability.

A common time base. Control rates differ by an order of magnitude, so trajectories are resampled to a shared frequency, and the policy usually emits an action chunk covering a fixed horizon rather than a single step, which makes the rate mismatch less damaging.

The generalist argument

The reason to bother is that skills decompose. Reaching, grasp approach, object identification and instruction following are largely embodiment-independent; only the last few centimetres of contact are specific. A model that learns "pick up the red bowl" from a hundred robots learns the semantics once, and the target robot's data only has to teach the contact.

This is also why co-training with non-robot data works. RT-2 co-fine-tunes a vision-language model on internet VQA data and robot trajectories by emitting actions as text tokens, and reports improved generalisation to novel objects, the ability to follow commands not present in the robot data, and rudimentary reasoning such as choosing a rock as an improvised hammer (Brohan et al., 2023, arXiv:2307.15818). OpenVLA makes the same bet in the open, a 7B model trained on 970k real demonstrations that beats the 55B RT-2-X by 16.5 points of absolute success across 29 tasks with 7× fewer parameters (Kim et al., 2024, arXiv:2406.09246).

When it breaks

Negative transfer is real and unevenly distributed. Pooling helps embodiments with little data and can hurt the ones with a lot, since the mixture pulls the policy toward the modal robot. Dataset mixture weights are a tuned hyperparameter, not an afterthought, and the weights that help a Franka arm are not the ones that help a mobile manipulator.

The action space is the bottleneck, not the model. End-effector deltas cannot express a five-finger regrasp or a whole-body motion. Any dexterity that does not survive projection into the shared space is invisible to the pooled model regardless of scale.

Sensor coverage is the lowest common denominator. Discarding force and torque to make datasets compatible removes exactly the signal that contact-rich tasks need. A pooled policy is systematically better at reaching than at inserting.

Evaluation does not pool. Every lab evaluates on its own hardware, with its own objects, lighting and reset procedure. Success rates from two institutions are not comparable, so the field's aggregate progress measure is weaker than its aggregate training set.