Low-Communication Distributed Training

Data-parallel training has one communication pattern and it is unforgiving: after every backward pass, every replica all-reduces the full gradient. For a 10B-parameter model in bf16 that is 20 GB moved per step, and at one step per second the fabric has to sustain tens of gigabytes per second between every pair of nodes. That requirement, more than any other, is why frontier training happens inside a single datacentre with a purpose-built interconnect.

DiLoCo asks what happens if replicas synchronise every 500 steps instead of every one. The answer on C4, with eight workers, was that it matched fully synchronous optimisation while communicating 500 times less (Douillard et al., 2023, arXiv:2311.08105).

The two-level optimiser

The algorithm is a variant of federated averaging with the hyperparameters pushed into a regime nobody had tried for language models. Each of \(M\) workers holds a full replica and runs \(H\) steps of ordinary AdamW on its own data shard, with no communication at all. After \(H\) steps, the workers form the average of their parameter deltas:

This quantity is treated as a pseudo-gradient and handed to a second, outer optimiser, which in DiLoCo is SGD with Nesterov momentum:

The result is broadcast and everyone continues. Two optimisers, two timescales: AdamW handles the fine-grained curvature of the loss inside a worker, and Nesterov momentum on the outer loop stabilises the aggregation of what are, by construction, stale and divergent updates.

The choice of outer optimiser is load-bearing rather than incidental. Plain averaging of the deltas, which is what FedAvg does (McMahan et al., AISTATS 2017, arXiv:1602.05629), works less well at this scale; the momentum term is what lets \(H\) be large. The theoretical ancestor is local SGD, whose convergence analysis showed that averaging occasionally still reaches mini-batch SGD rates while cutting communication rounds substantially (Stich, ICLR 2019, arXiv:1805.09767).

What the follow-up work established

Three results turned a promising algorithm into a deployable one.

It scales predictably. A scaling-law study under fixed compute budgets found DiLoCo scales both predictably and robustly with model size, and when well tuned scales better than data-parallel training, including larger optimal batch sizes and improved evaluation loss at a fixed token budget (Charles et al., 2025, arXiv:2503.09799). The framing shifted from "an acceptable degradation for bandwidth-poor settings" to "sometimes simply better".

Peak bandwidth can be flattened. Plain DiLoCo has a bursty profile: nothing for 500 steps, then the entire parameter set at once. Streaming DiLoCo synchronises parameter subsets in sequence rather than all at once, lets training continue during synchronisation, and quantises what is exchanged, reporting a further reduction of about two orders of magnitude in required bandwidth at billion-parameter scale (Douillard et al., 2025, arXiv:2501.18512).

It works on the open internet. OpenDiLoCo trained across two continents and three countries at 90 to 95% compute utilisation, with FP16 gradient reduction showing no performance loss (Jaghouar et al., 2024, arXiv:2407.07852). INTELLECT-1 then trained a 10B model on 1T tokens using up to 14 concurrent nodes across 3 continents and 30 independent compute providers joining and leaving during the run, holding 83 to 96% compute utilisation and 36.2 to 41.4% model FLOPs utilisation at a reported 400x reduction in communication bandwidth (Jaghouar et al., 2024, arXiv:2412.01152).

When it breaks

Memory is unchanged. Each worker holds a full replica plus full optimiser state. DiLoCo relaxes the bandwidth requirement, not the capacity requirement, so it composes with FSDP or tensor parallelism inside a worker rather than replacing them. A model that does not fit on an island still does not fit.

Straggler dynamics replace bandwidth limits. The synchronisation barrier is rarer, so each one costs more. One slow worker stalls every other worker for the full duration of the outer step, and heterogeneous hardware makes this the dominant inefficiency once bandwidth is no longer the bottleneck.

The inner-step count is a real hyperparameter. Too small and the communication saving disappears; too large and replicas drift far enough apart that averaging their deltas destroys progress. The workable range depends on model size, batch size, and data heterogeneity, and it has to be found empirically.

Verification becomes a problem when workers are untrusted. A permissionless swarm cannot assume workers computed what they claim. INTELLECT-2 addressed this explicitly with a verification protocol for untrusted worker outputs alongside its decentralised RL framework (Prime Intellect Team, 2025, arXiv:2505.07291). Closed clusters have no such need, and this is the clearest cost of opening the pool.