Training Dynamics & Scaling
Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.
11concepts
53flashcards
94minutes of reading
- 01 Loss Spikes and Divergence On almost every long pretraining run the smooth descending loss curve suddenly lurches upward, and diagnosing and recovering from these spikes is still closer to an operational skill than a solved engineering problem.
- 02 Scaling Laws and the Chinchilla Correction How loss falls predictably with compute, parameters, and data, and why the Chinchilla result showed almost every large model of its era was badly undertrained.
- 03 The Critical Batch Size Past a point that can be measured but not derived from first principles, adding more GPUs to widen the batch stops buying faster convergence per token, and just burns compute for redundant gradient information.
- 04 Warmup-Stable-Decay Schedules Cosine decay bakes the total step count into its formula before training starts; warmup-stable-decay schedules split the learning-rate curve so that commitment can be made at the very end instead.
- 05 muP and Hyperparameter Transfer The learning rate that is optimal for a 40-million-parameter proxy model is usually wrong for the 70-billion-parameter model it was meant to stand in for, and muP is the parametrisation designed to make that transfer actually work.