Robotics & Embodied AI
Vision-language-action models, action tokenisation, diffusion policies and sim-to-real.
10concepts
122flashcards
80minutes of reading
- 01 Cross-Embodiment Transfer Why robot data has never had an ImageNet, how Open X-Embodiment pooled 22 robot types into one training set, and what has to be true for data from one arm to help a different arm.
- 02 Flow-Matching Action Experts Why binning continuous robot actions into text tokens caps control frequency, and how attaching a flow-matching action head to a vision-language model reaches 50 Hz dexterous control.
- 03 RT-2 and Web-Scale Robot Learning RT-2 co-trains one transformer on internet vision-language data and robot trajectories by encoding actions as text tokens, transferring semantic web knowledge into robotic control.
- 04 Vision-Language-Action Models Turning a pretrained vision-language model into a robot policy that maps camera images plus a language instruction to motor actions, so the robot inherits web-scale semantic knowledge it could never learn from robot data alone.
- 05 World Models for Control How learning a latent dynamics model lets an agent train inside its own imagination, why that is the answer to robotics' sample-efficiency problem, and where the learned simulator's errors get exploited.