Distributed Training
Data, tensor and pipeline parallelism, ZeRO/FSDP sharding, mixed precision, and offload.
13concepts
165flashcards
111minutes of reading
- 01 Data Parallelism and DDP How replicating the model and sharding the batch across GPUs scales training, and why AllReduce is the primitive every framework eventually depends on.
- 02 Fault Tolerance at Scale Why a 16,000-GPU run fails every few hours, how checkpoint interval trades wasted compute against write cost, and what asynchronous and in-memory checkpointing changed.
- 03 Gradient Checkpointing, Activation Recomputation, and CPU Offload Why activations - not weights - usually dominate training memory, and how recomputation and CPU/NVMe offload trade compute and bandwidth to fit larger models.
- 04 Mixed-Precision Training (FP16, BF16, FP8) How lower-precision formats halve memory and double throughput on tensor cores, why BF16 displaced FP16 for training, and what FP8 changes on H100 and Blackwell.