Overlapping Communication and Computation

When \(t_{\text{comm}}(\text{layer}) \le t_{\text{compute}}(\text{work still to run})\). Compute per layer scales with batch size and hidden dimension; communication scales with bytes over link bandwidth. So small batches expose communication because there is too little compute to hide behind, and slow interconnects expose it from the other side. The same code can hit very different utilisation on NVLink versus Ethernet for exactly this reason.

Bucket size trades collective efficiency against how much compute remains to hide behind. Small buckets launch early but are latency-bound and waste bandwidth. Large buckets use the network efficiently but the final bucket fires when little backward compute is left, so it is exposed. The 25 MB default was tuned for models far smaller than current ones, and a model with very large individual weight tensors can defeat the pipelining entirely.

ZeRO-3 shards parameters, so each layer's weights must be all-gathered before it can run and are freed afterwards. Without prefetching layer \(L+1\)'s gather during layer \(L\)'s compute, every single layer becomes a synchronous stall and the step degenerates into alternating communication and computation. Prefetch depth is what makes full parameter sharding competitive, and it costs memory for the in-flight parameters.

With accumulation, most micro-steps skip the gradient all-reduce entirely and one step in \(k\) carries the whole collective. That reducing step has only its own backward compute to hide a full model's worth of gradient traffic, so overlap tuned against the common case is exposed exactly where it matters. The reducing step, not the average step, is the one to profile.

Exposed communication, not slow kernels. Well-tuned large runs land around 35 to 50 percent MFU, so 15 percent usually means collectives are running with no concurrent compute beneath them. Confirm with a profiler timeline showing NCCL kernels against gaps in the compute stream, then identify which of the four overlap sites (gradient reduce, parameter gather, tensor-parallel collective, pipeline transfer) is stalling.

When \(t_{\text{comm}}(\text{layer}) \le t_{\text{compute}}(\text{work still to run})\). Compute per layer scales with batch size and hidden dimension; communication scales with bytes over link bandwidth. So small batches expose communication because there is too little compute to hide behind, and slow interconnects expose it from the other side. The same code can hit very different utilisation on NVLink versus Ethernet for exactly this reason.

Bucket size trades collective efficiency against how much compute remains to hide behind. Small buckets launch early but are latency-bound and waste bandwidth. Large buckets use the network efficiently but the final bucket fires when little backward compute is left, so it is exposed. The 25 MB default was tuned for models far smaller than current ones, and a model with very large individual weight tensors can defeat the pipelining entirely.

ZeRO-3 shards parameters, so each layer's weights must be all-gathered before it can run and are freed afterwards. Without prefetching layer \(L+1\)'s gather during layer \(L\)'s compute, every single layer becomes a synchronous stall and the step degenerates into alternating communication and computation. Prefetch depth is what makes full parameter sharding competitive, and it costs memory for the in-flight parameters.

With accumulation, most micro-steps skip the gradient all-reduce entirely and one step in \(k\) carries the whole collective. That reducing step has only its own backward compute to hide a full model's worth of gradient traffic, so overlap tuned against the common case is exposed exactly where it matters. The reducing step, not the average step, is the one to profile.

Exposed communication, not slow kernels. Well-tuned large runs land around 35 to 50 percent MFU, so 15 percent usually means collectives are running with no concurrent compute beneath them. Confirm with a profiler timeline showing NCCL kernels against gaps in the compute stream, then identify which of the four overlap sites (gradient reduce, parameter gather, tensor-parallel collective, pipeline transfer) is stalling.