Training Dynamics & Scaling
Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.
11concepts
53flashcards
94minutes of reading
- 01 Gradient Accumulation The trick that lets a single GPU simulate a batch size far larger than what fits in its memory, at the cost of wall-clock time rather than compute.
- 02 Gradient Clipping A five-character config value, clip norm 1.0, appears in nearly every published pretraining recipe, and it is a crude safety net rather than a fix, one that is easy to misread as more protective than it is.
- 03 Learning-Rate Schedules The shape of the learning-rate curve across a training run matters as much as its peak value, and getting the shape wrong wastes a slice of a compute budget that was never coming back.
- 04 Optimiser State and Memory A model's raw weight size is only the starting point for training memory; Adam alone roughly quadruples the footprint before a single activation has been stored.
- 05 Warmup and Why It Helps The first few hundred to few thousand steps of a large training run are its most fragile, and a short linear ramp on the learning rate is the cheapest insurance against wrecking them.
- 06 Weight Decay in Pretraining Pretraining rarely revisits the same token twice, so the classical overfitting story for weight decay barely applies, yet nearly every LLM recipe still sets it to a nonzero value close to 0.1.
- 07 Loss Spikes and Divergence On almost every long pretraining run the smooth descending loss curve suddenly lurches upward, and diagnosing and recovering from these spikes is still closer to an operational skill than a solved engineering problem.
- 08 Scaling Laws and the Chinchilla Correction How loss falls predictably with compute, parameters, and data, and why the Chinchilla result showed almost every large model of its era was badly undertrained.
- 09 The Critical Batch Size Past a point that can be measured but not derived from first principles, adding more GPUs to widen the batch stops buying faster convergence per token, and just burns compute for redundant gradient information.
- 10 Warmup-Stable-Decay Schedules Cosine decay bakes the total step count into its formula before training starts; warmup-stable-decay schedules split the learning-rate curve so that commitment can be made at the very end instead.
- 11 muP and Hyperparameter Transfer The learning rate that is optimal for a 40-million-parameter proxy model is usually wrong for the 70-billion-parameter model it was meant to stand in for, and muP is the parametrisation designed to make that transfer actually work.