Training & Alignment
24 min
The Bandwidth Wall: How Low-Communication Training Unbundled the Datacentre
Data-parallel training all-reduces the entire gradient after every step, which is why frontier pretraining happens inside one building with a purpose-built fabric. DiLoCo synchronises every five hundred steps instead of every one…