Distributed Training
Data, tensor and pipeline parallelism, ZeRO/FSDP sharding, mixed precision, and offload.
13concepts
165flashcards
111minutes of reading
- 01 Data Parallelism and DDP How replicating the model and sharding the batch across GPUs scales training, and why AllReduce is the primitive every framework eventually depends on.
- 02 Fault Tolerance at Scale Why a 16,000-GPU run fails every few hours, how checkpoint interval trades wasted compute against write cost, and what asynchronous and in-memory checkpointing changed.
- 03 Gradient Checkpointing, Activation Recomputation, and CPU Offload Why activations - not weights - usually dominate training memory, and how recomputation and CPU/NVMe offload trade compute and bandwidth to fit larger models.
- 04 Mixed-Precision Training (FP16, BF16, FP8) How lower-precision formats halve memory and double throughput on tensor cores, why BF16 displaced FP16 for training, and what FP8 changes on H100 and Blackwell.
- 05 Composing Parallelism Strategies How data, tensor, pipeline, context and expert parallelism combine into one device mesh, why the ordering follows the interconnect hierarchy, and what each axis costs.
- 06 Expert Parallelism and All-to-All Sparse mixture-of-experts models are sharded by placing different experts on different devices, which replaces the all-reduce of dense training with two all-to-all exchanges per layer and makes routing a network-topology problem.
- 07 Gradient Compression and Quantised Collectives Ninety-nine point nine percent of the values in a distributed SGD gradient exchange are redundant, and twenty years of compression research shows that exploiting this is easy in theory and constrained in practice by one property: whether the compressed form survives an all-reduce.
- 08 Low-Communication Distributed Training Standard data parallelism synchronises gradients every step, which requires a datacentre-grade fabric; DiLoCo synchronises every few hundred steps instead and trains language models across poorly connected islands of accelerators.
- 09 Overlapping Communication and Computation Why a large training run's collectives are mostly free when overlapped and catastrophic when exposed, and the bucketing, prefetch and scheduling tricks that hide them.
- 10 Pipeline Bubbles and Schedules Why pipeline parallelism wastes device time by construction, how the bubble fraction depends on micro-batch count, and what 1F1B, interleaving and zero-bubble schedules recover.
- 11 Sequence and Context Parallelism Why long-context training runs out of memory even when the model fits, and how Ring Attention and Ulysses split the sequence dimension across devices without changing the maths.
- 12 Tensor and Pipeline Parallelism How frontier labs split a model across thousands of GPUs by sharding within layers (tensor parallel) and across layers (pipeline parallel), and how to pick the split.
- 13 ZeRO and FSDP How sharding optimiser state, gradients, and parameters across data-parallel ranks turns a memory problem into a bandwidth problem, and why FSDP is now the PyTorch default.