The Bandwidth Wall: How Low-Communication Training Unbundled the Datacentre

Work out what standard data-parallel training actually asks of a network. A 10-billion-parameter model in bf16 has a 20 GB gradient. Ring all-reduce moves close to twice that per rank, so across eight workers each one pushes 35 GB through the fabric every step. If a step takes 1.5 seconds, the sustained per-rank requirement is 23.3 GB/s, which is 187 Gbps, and it has to hold for the duration of the run.

That number is why frontier pretraining happens in one building. It is not a preference for co-location; it is a hard constraint, and it has shaped the industry's capital structure. Compute that exists but is not connected by an InfiniBand-class fabric has been, for practical purposes, unusable for training.

In November 2023 a Google DeepMind team asked what happens if you simply stop synchronising so often. Each worker runs 500 optimiser steps entirely alone, then the workers exchange averaged parameter deltas rather than gradients. On C4 with eight workers, this matched fully synchronous optimisation while communicating 500 times less (Douillard et al., 2023, DiLoCo, arXiv:2311.08105).

Why this matters: The bandwidth requirement of data parallelism determines where models can be trained, who can train them, and what a training cluster costs. If the synchronisation interval is a tunable parameter rather than a fixed property of the algorithm, then heterogeneous, geographically scattered, intermittently available compute becomes viable, and the set of organisations able to pretrain a frontier-scale model changes.

TL;DR

• Standard data parallelism all-reduces the full gradient every step. Relaxing the frequency by a factor \(H\) relaxes the average bandwidth requirement by roughly the same factor, which is where the headline 500x comes from.
• DiLoCo is a two-level optimiser: AdamW inside each worker for \(H\) steps, then Nesterov-momentum SGD outside on the averaged parameter delta. Both levels are load-bearing; plain averaging works less well.
• The exchanged quantity is not a gradient. It is the accumulated effect of hundreds of local steps, and treating it as a pseudo-gradient with momentum is what allows \(H\) to be large.
• A fixed-compute scaling study found DiLoCo scales predictably, and when well tuned scales better than data-parallel training, with larger optimal batch sizes and better evaluation loss at a fixed token budget.
• INTELLECT-1 trained a 10B model on 1T tokens across 3 continents and 30 independent compute providers joining and leaving mid-run, holding 83 to 96% compute utilisation and 36.2 to 41.4% MFU at a reported 400x bandwidth reduction.
• Streaming DiLoCo attacks peak rather than average bandwidth by synchronising parameter fragments in sequence, overlapping the exchange with training, and quantising it, reporting a further reduction of about two orders of magnitude.
• Nothing here reduces memory. Every worker still holds a full replica plus optimiser state, so DiLoCo composes with FSDP and tensor parallelism rather than replacing them.
• The binding constraint moves rather than disappearing: with rare synchronisation, stragglers and worker verification become the dominant costs.

At a Glance

flowchart LR
    subgraph inner["Inner loop, no communication"]
        W1["Worker 1<br/>AdamW, H steps"]
        W2["Worker 2<br/>AdamW, H steps"]
        W3["Worker M<br/>AdamW, H steps"]
    end
    W1 --> A["Average the deltas"]
    W2 --> A
    W3 --> A
    A --> O["Outer optimiser<br/>Nesterov momentum"]
    O --> B["Broadcast new weights"]
    B -.->|"next outer step"| inner
    O --> C["Bandwidth: one exchange<br/>per H steps"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class W1,W2,W3 blue
    class A,O purple
    class B teal
    class C emerald

The structure is two nested loops with completely different time constants. Everything expensive about the network happens in the outer one, and the outer one runs a few hundred times less often than the inner.

Before the Outer Loop

The idea is not new. Averaging occasionally rather than synchronising constantly is the core of federated learning, proposed for a setting where the network was a mobile phone's uplink and the data could not leave the device. McMahan and colleagues reported 10 to 100 times fewer communication rounds than synchronous SGD across five architectures and four datasets, on unbalanced and non-IID data (McMahan et al., AISTATS 2017, arXiv:1602.05629).

The theory followed. Stich analysed local SGD, where workers run independently and average only occasionally, and proved it matches mini-batch SGD in gradient evaluations while cutting communication rounds by a factor of \(T^{1/2}\), with linear speedup in worker count on convex problems (Stich, ICLR 2019, arXiv:1805.09767).

Meanwhile a separate research programme attacked the same bottleneck from the other side, shrinking the payload rather than the frequency. Deep Gradient Compression found 99.9% of the gradient exchange redundant and reported 270x to 600x compression with no accuracy loss (Lin et al., ICLR 2018, arXiv:1712.01887). PowerSGD then solved the problem that had blocked deployment, producing a compressed form that survives an all-reduce because rank-\(r\) factors can be summed directly (Vogels et al., NeurIPS 2019, arXiv:1905.13727).

Neither line was applied to language-model pretraining at scale. Federated learning was understood as a privacy technology for edge devices; gradient compression was understood as a fix for clusters with weak interconnects. The conceptual step DiLoCo took was to treat the two as the same problem, apply the federated recipe to LLM pretraining, and push the inner-step count to a value nobody in the federated literature had tried.

timeline
    title From federated averaging to decentralised pretraining
    2016 : McMahan et al. propose federated averaging for edge devices
         : 10 to 100 times fewer communication rounds than synchronous SGD
    2018 : Stich proves local SGD converges fast and communicates little
         : Deep Gradient Compression reports 270x to 600x on the payload
    2019 : PowerSGD makes compression survive the all-reduce via low-rank factors
    2023 : DiLoCo pairs inner AdamW with outer Nesterov momentum
         : Matches synchronous training at 500 times less communication
    2024 : OpenDiLoCo trains across two continents at 90 to 95% compute utilisation
         : INTELLECT-1 trains 10B on 1T tokens across 30 independent providers
    2025 : Streaming DiLoCo cuts peak bandwidth by roughly two more orders of magnitude
         : Scaling laws show DiLoCo beating data parallelism when well tuned
         : INTELLECT-2 extends the approach to decentralised reinforcement learning

[IMAGE: Two bandwidth-versus-time traces for the same training run. The top, labelled DDP, is a dense sawtooth at 26.7 GB per second sustained. The bottom, labelled DiLoCo H equals 500, is flat at zero with a single narrow spike every 750 seconds. Caption: "Same total work, two completely different demands on the network."]

How DiLoCo Actually Works

Two optimisers on two timescales

Each of \(M\) workers holds a full replica \(\theta_m\) and its own data shard. For \(H\) steps it runs ordinary AdamW locally, exchanging nothing. At the end of the interval, the workers form the average of their parameter deltas:

This quantity is handed to an outer optimiser as if it were a gradient:

and in DiLoCo the outer optimiser is SGD with Nesterov momentum. The result is broadcast, every worker adopts it, and the next interval begins.

Three details in that description do the work.

The exchanged object is not a gradient. A gradient is a local linearisation valid in a neighbourhood of one point. \(\Delta^{(t)}\) is the net displacement produced by 500 AdamW steps, each of which moved somewhere the others did not see. Calling it a pseudo-gradient is a convenience, and the fact that outer optimisation on it works at all is the empirical content of the paper.

The outer optimiser is not incidental. Plain averaging, which is what FedAvg does, is the special case where the outer optimiser is SGD with learning rate 1 and no momentum, and it performs worse at this scale. Momentum on the outer loop damps the disagreement between replicas, which is precisely what accumulates as \(H\) grows, and it is what makes a 500-step interval viable rather than a 5-step one.

Two optimisers, two jobs. AdamW inside handles the ill-conditioned local curvature of a transformer loss. Nesterov outside handles a low-dimensional, high-variance aggregation problem. Using AdamW in both places is worse, which is a hint about what the outer problem actually is.

[IMAGE: A two-dimensional loss-landscape sketch with a shared starting point and eight trajectories fanning out over 500 local steps, their endpoints ringed by a dashed circle, and a single arrow from the start to the circle's centroid labelled "averaged delta". Caption: "Replicas separate, and over a few hundred steps the average of their displacements is still a sensible direction."]

Why the interval can be so long

The intuition that fails here is that replicas will diverge into different basins. In practice, over a few hundred steps from a shared initialisation with a shared data distribution, they stay in a region where the average of their displacements is a sensible direction. The paper also reports strong robustness to the data distribution of each worker, meaning the shards do not need to be IID, and robustness to workers appearing and disappearing during training.

That last property is not a footnote. Under synchronous data parallelism a lost worker halts the step. Under DiLoCo the remaining workers are mid-interval and unaffected, and the loss is a smaller \(M\) in the next average.

Attacking peak bandwidth, not just average

Plain DiLoCo has a bursty profile: nothing for 500 steps, then the entire parameter set at once. Average bandwidth falls by \(H\); peak bandwidth does not fall at all, and the burst stalls training while it drains.

Streaming DiLoCo makes three changes. Parameter subsets are synchronised in sequence rather than the whole model at once, which spreads the burst. Training continues during synchronisation, so the exchange overlaps with compute instead of blocking it. And the exchanged data is quantised. Together these enable billion-parameter training with a further reduction of about two orders of magnitude in required bandwidth, at comparable quality (Douillard et al., 2025, arXiv:2501.18512).

Overlapping is the subtle one. It means the outer update is applied to weights that have already moved, so the algorithm is deliberately accepting staleness in exchange for removing a stall. That is a familiar tradeoff from asynchronous SGD, and here it is bounded, because the staleness is at most one outer interval.

Does it actually scale

The obvious worry about any method validated at small scale is that the gap reopens as models grow. Charles and colleagues ran the fixed-compute scaling study and found the opposite of the worry. DiLoCo scales predictably and robustly with model size; when well tuned it scales better than data-parallel training and can beat it even at small model sizes; and the benefits include increased optimal batch sizes, improved downstream generalisation with scale, and better evaluation loss at a fixed token budget (Charles et al., 2025, arXiv:2503.09799).

The larger optimal batch size is the most useful result for a practitioner, because batch size is itself a scaling constraint: data parallelism at extreme worker counts pushes past the critical batch size and wastes compute. A method that raises that ceiling is buying something orthogonal to bandwidth.

[IMAGE: Loss-versus-tokens curves for data-parallel and DiLoCo at three model scales, drawn as three panels. The gap between the two curves narrows and inverts as scale increases, with an annotation at the largest panel reading "DiLoCo ahead when tuned". Caption: "The worry was that the gap widens with scale. The measurement says it does not."]

Seeing It in Motion

sequenceDiagram
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant NET as Fabric
    participant OO as Outer optimiser
    Note over W1,W2: interval t begins from shared weights
    loop H local AdamW steps
        W1->>W1: forward, backward, update
        W2->>W2: forward, backward, update
    end
    W1->>NET: parameter delta
    W2->>NET: parameter delta
    NET->>OO: averaged delta
    OO->>OO: Nesterov momentum step
    OO->>NET: new shared weights
    NET->>W1: broadcast
    NET->>W2: broadcast
    Note over W1,W2: network idle for the next H steps

The idle note is the entire economic argument. For 500 steps the fabric carries nothing, which means the fabric can be a commodity internet link rather than a dedicated interconnect.

graph TD
    Q{"What is scarce?"}
    Q -->|"inter-node bandwidth"| D["DiLoCo family"]
    Q -->|"device memory"| F["FSDP or ZeRO sharding"]
    Q -->|"both"| C["Compose: DiLoCo outside,<br/>FSDP inside each island"]
    Q -->|"neither"| P["Standard DDP"]
    D --> S1["Peak bandwidth still bursty"]
    S1 --> S2["Streaming DiLoCo:<br/>fragments, overlap, quantise"]
    C --> S3["Island equals one datacentre"]
    F --> S4["Memory solved, bandwidth unchanged"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class Q slate
    class D,F purple
    class C teal
    class P blue
    class S1,S2,S3,S4 amber

The decision tree matters because the two constraints are routinely conflated. Sharding solves memory and leaves bandwidth untouched; DiLoCo solves bandwidth and leaves memory untouched. A run that is short of both needs both, arranged as DiLoCo between islands and FSDP within them.

By the Numbers

Sources: McMahan et al. (2017), Stich (2019), Lin et al. (2018), Douillard et al. (2023), Jaghouar et al. (2024a), Jaghouar et al. (2024b), Douillard et al. (2025), Charles et al. (2025), Prime Intellect Team (2025). Compression and bandwidth-reduction factors are each measured against that paper's own baseline and are not directly comparable across rows.

A Concrete Example

The bandwidth arithmetic, worked end to end for a 10B-parameter model, so the 500x is derived rather than quoted.

Setup. 10B parameters, bf16, so the parameter and gradient tensors are 20 GB each. Assume 8 workers, an achieved step time of 1.5 seconds, and ring all-reduce.

Step 1: bytes per rank per all-reduce. Ring all-reduce is two passes, reduce-scatter then all-gather, each moving \(\frac{M-1}{M} \cdot S\) bytes per rank. Total per rank:

Step 2: DDP sustained requirement. One all-reduce per 1.5-second step:

This needs a datacentre fabric. A good 1 Gbps internet link supplies 0.125 GB/s, which is 186 times short. Training would spend 280 seconds communicating for every 1.5 seconds computing.

Step 3: DiLoCo with H equals 500. One exchange of the same 35 GB per rank, but the interval is now \(500 \times 1.5 = 750\) seconds of compute:

That fits inside a 1 Gbps link with headroom. The ratio to step 2 is \(23.3 / 0.0467 = 500\), which is \(H\) exactly, and that is where the paper's headline number comes from: it is not an empirical constant, it is the synchronisation interval.

Step 4: the part the average hides. The 373 Mbps is an average over 750 seconds. The actual transfer is 35 GB delivered as fast as the link allows. At 1 Gbps that is 280 seconds during which training stalls, so the effective compute utilisation is \(750 / (750 + 280) = 73\%\). The bandwidth problem is solved and a 27% stall remains.

Step 5: what Streaming DiLoCo removes. Split the parameters into 10 fragments synchronised at staggered offsets, and each transfer is 3.5 GB over 28 seconds. Let training continue during the transfer, and the stall becomes zero at the price of applying an outer update to weights that have moved. Quantise the exchange from bf16 to int8 and each fragment halves again. Utilisation returns to the high nineties on the same link.

What the arithmetic shows. DiLoCo converts a bandwidth problem into a latency-tolerance problem. Step 3 is the headline; step 4 is why Streaming DiLoCo exists; step 5 is why the second paper reports gains on top of a result that already looked complete.

[IMAGE: A stacked-time bar for one 750-second outer interval under three schemes. Plain DiLoCo shows 750 s compute plus a 280 s red stall block; Streaming DiLoCo shows ten small transfer blocks overlaid on the compute bar in a lighter shade; DDP shows a compute bar swamped by transfer. Caption: "Average bandwidth is solved by the interval; peak bandwidth is solved by fragmenting and overlapping."]

Where It Breaks

Memory is untouched, and it is usually the first wall

Every worker holds a full parameter replica plus AdamW's two moment buffers. For a 10B model in bf16 with fp32 optimiser state, that is on the order of 140 GB per replica before activations. DiLoCo says nothing about this. A model that does not fit on an island still does not fit, and the honest description of the method is that it relaxes the constraint between islands while leaving the constraint within them exactly where it was.

Stragglers replace bandwidth as the binding cost

Fewer barriers means each barrier is more expensive. A worker that is 20% slower does not lose 20% of one step; it holds every other worker at the outer boundary for 20% of a 750-second interval. On homogeneous hardware this is a scheduling nuisance. On a swarm of mixed GPUs across three continents it is the dominant inefficiency, which is why decentralised runs report compute utilisation as their headline efficiency number rather than bandwidth.

[IMAGE: A Gantt chart of one outer interval across eight workers with mismatched bar lengths, the fastest finishing at 620 seconds and the slowest at 900, with the idle gap shaded amber and labelled "seven workers waiting". Caption: "Rare barriers make each barrier expensive; a 20 percent slower worker costs 20 percent of an entire interval."]

The inner-step count is a genuine hyperparameter with no theory

Too small and the communication saving evaporates. Too large and the replicas drift far enough that averaging their displacements destroys progress. The workable range depends on model size, batch size, learning-rate schedule, and how heterogeneous the shards are, and it is found empirically. The scaling-law work characterises behaviour across model sizes but does not produce a formula that gives you \(H\).

Overlapped synchronisation is deliberate staleness

Streaming DiLoCo's overlap means the outer update lands on weights that have already moved during the transfer. That is asynchronous optimisation with a bounded delay, and it inherits asynchronous optimisation's sensitivity: the tolerable staleness depends on the outer learning rate, and the failure mode is a slow quality regression rather than a crash.

Untrusted workers need a verification layer

A private cluster assumes workers computed what they report. A permissionless swarm cannot, and a single malicious or broken worker contributes a delta to an average that everyone then adopts. INTELLECT-2 shipped an explicit protocol for verifying untrusted worker outputs alongside its decentralised RL framework (Prime Intellect Team, 2025, arXiv:2505.07291). This is a real cost of opening the pool and it has nothing to do with optimisation.

Debugging loses its usual footholds

With synchronous training, every rank has identical weights at every step, and a divergence between ranks is unambiguously a bug. Under DiLoCo, ranks legitimately hold different weights for hundreds of steps, so the cheapest consistency check available to a distributed-training engineer is gone. Loss curves are per-worker and noisy between synchronisations, and an unhealthy worker looks like a worker with a harder data shard.

The reinforcement-learning setting is harder than pretraining

Pretraining has a fixed dataset and a stationary objective. RL has a policy that changes, rollouts generated by that policy, and a strong dependence on how stale the generating policy is. Extending low-communication methods to RL required additional machinery for policy distribution and verification rather than a straight port, and the design space there is much less settled than for pretraining.

Alternative Designs

The row that matters most for the confusion in the field is the second. ZeRO and DiLoCo are frequently discussed as alternatives and they are orthogonal: sharding increases communication to reduce memory, DiLoCo reduces communication and leaves memory alone. The realistic large-scale configuration uses both.

How It Is Used in Practice

Two organisations have driven the public work and they have different reasons.

Google DeepMind's interest reads as strategic rather than resource-constrained. The original DiLoCo paper opens by observing that it is difficult to build and maintain a single cluster hosting many accelerators, and easier to find several smaller ones. That is an argument about the practical ceiling on cluster size and about scheduling flexibility across a fleet, not about lacking bandwidth. The scaling-law follow-up, which shows better batch-size scaling, points at the same motivation: the method is being evaluated as a default, not as a fallback.

Prime Intellect's interest is the decentralised one. OpenDiLoCo reproduced and extended the method in the open, training across two continents and three countries at 90 to 95% compute utilisation and confirming that FP16 gradient reduction costs nothing (Jaghouar et al., 2024, arXiv:2407.07852). INTELLECT-1 then ran the full experiment: a 10B model, 1T tokens, up to 14 concurrent nodes across 3 continents, 30 independent providers joining and leaving during the run, 83 to 96% compute utilisation and 36.2 to 41.4% MFU (Jaghouar et al., 2024, arXiv:2412.01152). INTELLECT-2 extended it to reinforcement learning on a 32B reasoning model with a permissionless swarm, adding verification of untrusted outputs and efficient policy distribution.

The engineering considerations that come with this are not the ones a single-cluster team is used to. Checkpointing has to tolerate a variable worker set. Data sharding has to be assigned to workers who may vanish. Cost accounting spans providers with different pricing. And the failure domain is the outer interval rather than the step, which changes what monitoring is useful: per-worker throughput and time-to-barrier matter more than aggregate FLOPS.

[IMAGE: A world map with three clusters of nodes on different continents, thick internal links and thin dashed links between continents, annotated with the per-rank bandwidth figures for DDP and DiLoCo on each link type. Caption: "The thin links are the ones the algorithm had to stop needing."]

[IMAGE: A grid with memory constraint on one axis and bandwidth constraint on the other, four quadrants labelled DDP, FSDP alone, DiLoCo alone, and DiLoCo wrapping FSDP islands, each cell carrying a one-line note on what it does and does not solve. Caption: "Two constraints, two mechanisms, and the configuration that needs both."]

Insights Worth Remembering

1. The 500x is the synchronisation interval, not a discovered constant. Average bandwidth falls by exactly \(H\) because the same payload is amortised over \(H\) times more compute. Knowing this tells you the number is a design choice you can move, and where the ceiling on moving it comes from.

2. The exchanged object is not a gradient and should not be reasoned about as one. It is the net displacement of hundreds of adaptive steps. That is why the outer optimiser needs momentum, why the outer learning rate is not the inner one, and why intuitions imported from gradient accumulation mislead.

3. Sharding and low-communication training solve opposite constraints. ZeRO and FSDP trade more communication for less memory. DiLoCo trades nothing for less communication and leaves memory alone. Treating them as competing options is the most common conceptual error in this area.

4. Solving average bandwidth exposes peak bandwidth. The burst that DiLoCo leaves behind can stall a run for a quarter of its time on a commodity link. Streaming DiLoCo exists because the first result was necessary and not sufficient, and the same pattern will recur for whatever the next binding constraint turns out to be.

5. Robustness to churn is a first-class property, not a bonus. Workers appearing and disappearing mid-run is fatal under synchronous data parallelism and merely changes \(M\) under DiLoCo. That single property is what makes permissionless training conceivable at all.

6. The constraint moves rather than vanishing. Once bandwidth stops binding, straggler dynamics and worker verification take over. A decentralised run's headline efficiency number is compute utilisation for exactly this reason, and that number is governed by the slowest participant.

7. Better batch-size scaling may end up mattering more than bandwidth. Data parallelism at extreme worker counts runs into the critical batch size and wastes compute. A method that raises the optimal batch size is buying something that a faster fabric cannot buy, and that is the finding most likely to make this a default rather than a niche.

Open Questions

Where is the ceiling on \(H\)? Empirically, several hundred inner steps works. There is no theory that predicts where it stops working as a function of model size, batch size, and shard heterogeneity, and the scaling-law study characterises behaviour without producing that prediction. An adaptive \(H\), driven by a measured divergence between replicas, is an obvious idea with no published treatment.

Can the outer optimiser be improved? Nesterov momentum was a reasonable first choice and has not been seriously challenged. The outer problem is low-dimensional, high-variance, and heavily correlated across intervals, which is a profile that ought to reward a purpose-built optimiser. Whether adaptive or second-order outer methods help is untested at scale.

Does the approach hold for reinforcement learning? INTELLECT-2 shows it can be made to work for a 32B reasoning model, and the RL setting adds policy staleness on top of weight staleness. How the two interact, and whether the interval that is safe for pretraining is safe for RL, is an open empirical question rather than a settled one.

What happens to interpretability and safety auditing of a model no single party trained? A model produced by 30 independent providers has a provenance chain that no existing audit methodology addresses. Data attribution, contamination checks, and backdoor detection all assume a known training pipeline. This is a governance question that the technical work has outrun.

Is any of this cheaper in practice? The papers report bandwidth reductions and compute utilisation, not total cost. Spot GPU capacity across many providers, with the coordination overhead and verification cost included, may or may not undercut a reserved cluster, and the comparison is heavily dependent on market conditions at the time it is made. No published study makes it honestly.

Sources and Further Reading

1. Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., & Shen, J. (2023). "DiLoCo: Distributed Low-Communication Training of Language Models." arXiv:2311.08105
2. Douillard, A., Donchev, Y., Rush, K., Kale, S., Charles, Z., Garrett, Z., Teston, G., Lacey, D., McIlroy, R., Shen, J., Ramé, A., Szlam, A., Ranzato, M., & Barham, P. (2025). "Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch." arXiv:2501.18512
3. Charles, Z., Teston, G., Dery, L., Rush, K., Fallen, N., Garrett, Z., Szlam, A., & Douillard, A. (2025). "Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo." arXiv:2503.09799
4. Jaghouar, S., Ong, J. M., & Hagemann, J. (2024). "OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training." arXiv:2407.07852
5. Jaghouar, S., Ong, J. M., Basra, M., Obeid, F., Straube, J., Keiblinger, M., Bakouch, E., Atkins, L., Panahi, M., Goddard, C., Ryabinin, M., & Hagemann, J. (2024). "INTELLECT-1 Technical Report." arXiv:2412.01152
6. Prime Intellect Team (2025). "INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning." arXiv:2505.07291
7. McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Agüera y Arcas, B. (2017). "Communication-Efficient Learning of Deep Networks from Decentralized Data." AISTATS 2017. arXiv:1602.05629
8. Stich, S. U. (2019). "Local SGD Converges Fast and Communicates Little." ICLR 2019. arXiv:1805.09767
9. Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2018). "Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training." ICLR 2018. arXiv:1712.01887
10. Vogels, T., Karimireddy, S. P., & Jaggi, M. (2019). "PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization." NeurIPS 2019. arXiv:1905.13727
11. Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnovic, M. (2017). "QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding." arXiv:1610.02132
12. Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018). "signSGD: Compressed Optimisation for Non-Convex Problems." arXiv:1802.04434
13. DeepSeek-AI (2024). "DeepSeek-V3 Technical Report." arXiv:2412.19437
14. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020). "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding." arXiv:2006.16668

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.