Massively Parallel Simulation and Sim-to-Real Locomotion
How putting the physics engine on the GPU turned weeks of legged-robot RL into minutes, and why domain randomisation is the only reason those policies survive contact with a real floor.
For most of the 2010s, reinforcement learning on legged robots was a cluster job. The physics engine ran on CPUs, the policy ran on a GPU, and every step of every environment shipped observations across the PCIe bus and back. The simulator, not the learning algorithm, was the bottleneck, and a quadruped locomotion policy was a multi-day affair on hundreds of cores.
Isaac Gym removed the transfer by keeping both sides on the device: physics state lives in GPU buffers that are handed to PyTorch as tensors with no copy, so thousands of environments step in parallel and the policy update reads them in place. The reported gain is 2 to 3 orders of magnitude over conventional RL training that pairs a CPU simulator with a GPU network (Makoviychuk et al., 2021, arXiv:2108.10470). Locomotion policies that had taken days now take minutes on a single workstation GPU.
Why parallelism changes the algorithm, not just the clock
With 4,096 environments running at once, an on-policy method collects an enormous, freshly sampled batch every iteration. That makes PPO-style algorithms behave far better than they do with 16 environments: gradient noise drops, the effective batch is large enough that fewer epochs per batch are needed, and short rollouts suffice because there are so many of them. The practical recipe inverts. Instead of long trajectories from few environments, take very short trajectories from very many.
It also changes what is affordable. Reward shaping, curricula over terrain difficulty, and randomisation ranges can all be searched empirically because a full training run is a coffee break. Much of the progress in legged locomotion since 2021 is downstream of that iteration speed rather than of any new algorithm.
The reality gap and how randomisation crosses it
A policy trained in one simulator learns that simulator's friction model, actuator response and contact solver, none of which match a real robot. Domain randomisation attacks this by refusing to let the policy rely on any of them: mass, friction, motor gains, latency, terrain and sensor noise are resampled every episode, so the only strategy that works across the distribution is one that works robustly. If the real robot's parameters lie inside the training distribution, the policy treats reality as one more sample.
Choosing that distribution by hand is the hard part, and automatic domain randomisation removes some of the guesswork by growing the range as the policy succeeds, generating environments of ever-increasing difficulty; OpenAI used it to transfer a Rubik's-cube-manipulating hand policy from simulation to real hardware, and observed memory-augmented policies showing signs of emergent meta-learning, adapting at test time to conditions they had never seen (OpenAI et al., 2019, arXiv:1910.07113).
Actuator dynamics deserve a separate mention. Analytical motor models were a major source of the gap for legged robots, and learning an actuator model from real hardware data and inserting it into the simulator was the key step in the first demonstrations of learned quadruped control transferring to real machines (Hwangbo et al., Science Robotics, 2019).
Where it breaks
Randomisation buys robustness by paying performance. A policy that must work across every friction coefficient in a wide range is conservative on all of them. Widening the distribution to guarantee transfer produces gaits that are visibly stiffer and less efficient than a policy tuned to the true dynamics.
Contact-rich manipulation does not transfer like locomotion. Legged locomotion is forgiving because the feedback loop is fast and errors are self-correcting. Insertion, deformable objects and multi-finger regrasping depend on contact forces that simulators model poorly, and no amount of randomising a wrong model produces the right behaviour.
Simulated success is not evidence. Reward hacking in simulation is routine: policies exploit solver artefacts, penetrate geometry, or ride numerical instabilities. Any sim result without hardware validation, or at minimum a simulator whose correlation with hardware has been measured, should be read as an upper bound.
12 flashcards for this concept
Click a card to reveal the answer.