Distributed Training
Data, tensor and pipeline parallelism, ZeRO/FSDP sharding, mixed precision, and offload.
10concepts
70flashcards
85minutes of reading
- 01 Composing Parallelism Strategies How data, tensor, pipeline, context and expert parallelism combine into one device mesh, why the ordering follows the interconnect hierarchy, and what each axis costs.
- 02 Overlapping Communication and Computation Why a large training run's collectives are mostly free when overlapped and catastrophic when exposed, and the bucketing, prefetch and scheduling tricks that hide them.
- 03 Pipeline Bubbles and Schedules Why pipeline parallelism wastes device time by construction, how the bubble fraction depends on micro-batch count, and what 1F1B, interleaving and zero-bubble schedules recover.
- 04 Sequence and Context Parallelism Why long-context training runs out of memory even when the model fits, and how Ring Attention and Ulysses split the sequence dimension across devices without changing the maths.
- 05 Tensor and Pipeline Parallelism How frontier labs split a model across thousands of GPUs by sharding within layers (tensor parallel) and across layers (pipeline parallel), and how to pick the split.
- 06 ZeRO and FSDP How sharding optimiser state, gradients, and parameters across data-parallel ranks turns a memory problem into a bandwidth problem, and why FSDP is now the PyTorch default.