Distributed Training
Data, tensor and pipeline parallelism, ZeRO/FSDP sharding, mixed precision, and offload.
13concepts
165flashcards
111minutes of reading
- 01 Composing Parallelism Strategies How data, tensor, pipeline, context and expert parallelism combine into one device mesh, why the ordering follows the interconnect hierarchy, and what each axis costs.
- 02 Expert Parallelism and All-to-All Sparse mixture-of-experts models are sharded by placing different experts on different devices, which replaces the all-reduce of dense training with two all-to-all exchanges per layer and makes routing a network-topology problem.
- 03 Gradient Compression and Quantised Collectives Ninety-nine point nine percent of the values in a distributed SGD gradient exchange are redundant, and twenty years of compression research shows that exploiting this is easy in theory and constrained in practice by one property: whether the compressed form survives an all-reduce.
- 04 Low-Communication Distributed Training Standard data parallelism synchronises gradients every step, which requires a datacentre-grade fabric; DiLoCo synchronises every few hundred steps instead and trains language models across poorly connected islands of accelerators.
- 05 Overlapping Communication and Computation Why a large training run's collectives are mostly free when overlapped and catastrophic when exposed, and the bucketing, prefetch and scheduling tricks that hide them.
- 06 Pipeline Bubbles and Schedules Why pipeline parallelism wastes device time by construction, how the bubble fraction depends on micro-batch count, and what 1F1B, interleaving and zero-bubble schedules recover.
- 07 Sequence and Context Parallelism Why long-context training runs out of memory even when the model fits, and how Ring Attention and Ulysses split the sequence dimension across devices without changing the maths.
- 08 Tensor and Pipeline Parallelism How frontier labs split a model across thousands of GPUs by sharding within layers (tensor parallel) and across layers (pipeline parallel), and how to pick the split.
- 09 ZeRO and FSDP How sharding optimiser state, gradients, and parameters across data-parallel ranks turns a memory problem into a bandwidth problem, and why FSDP is now the PyTorch default.