Training Dynamics & Scaling
Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.
16concepts
157flashcards
135minutes of reading
- 01 Gradient Accumulation The trick that lets a single GPU simulate a batch size far larger than what fits in its memory, at the cost of wall-clock time rather than compute.
- 02 Gradient Clipping A five-character config value, clip norm 1.0, appears in nearly every published pretraining recipe, and it is a crude safety net rather than a fix, one that is easy to misread as more protective than it is.
- 03 Learning-Rate Schedules The shape of the learning-rate curve across a training run matters as much as its peak value, and getting the shape wrong wastes a slice of a compute budget that was never coming back.
- 04 Optimiser State and Memory A model's raw weight size is only the starting point for training memory; Adam alone roughly quadruples the footprint before a single activation has been stored.
- 05 Warmup and Why It Helps The first few hundred to few thousand steps of a large training run are its most fragile, and a short linear ramp on the learning rate is the cheapest insurance against wrecking them.
- 06 Weight Decay in Pretraining Pretraining rarely revisits the same token twice, so the classical overfitting story for weight decay barely applies, yet nearly every LLM recipe still sets it to a nonzero value close to 0.1.
- 07 Double Descent Test error rises to a peak at the interpolation threshold and then falls again as models grow past it, which breaks the U-shaped bias-variance picture and explains why more parameters can be safe.
- 08 Emergent Abilities and Metric Artefacts Capabilities that appear to switch on abruptly at a critical scale, the argument that the abruptness comes from the metric rather than the model, and what survives the critique.
- 09 Grokking and Delayed Generalisation Networks that memorise a small dataset perfectly and then, thousands of steps later, suddenly generalise, why the delay happens, and what the four competing explanations agree on.
- 10 Loss Spikes and Divergence On almost every long pretraining run the smooth descending loss curve suddenly lurches upward, and diagnosing and recovering from these spikes is still closer to an operational skill than a solved engineering problem.
- 11 Scaling Laws and the Chinchilla Correction How loss falls predictably with compute, parameters, and data, and why the Chinchilla result showed almost every large model of its era was badly undertrained.
- 12 The Critical Batch Size Past a point that can be measured but not derived from first principles, adding more GPUs to widen the batch stops buying faster convergence per token, and just burns compute for redundant gradient information.
- 13 The Edge of Stability Gradient descent on neural networks does not stay in the regime where its convergence theory applies; the loss curvature grows until training is marginally unstable, and then hovers there.
- 14 The Neural Tangent Kernel and Feature Learning In the infinite-width limit a network trains like a fixed kernel machine, which makes it analysable and also strictly weaker than real networks, and the gap between the two regimes is where deep learning's advantage lives.
- 15 Warmup-Stable-Decay Schedules Cosine decay bakes the total step count into its formula before training starts; warmup-stable-decay schedules split the learning-rate curve so that commitment can be made at the very end instead.
- 16 muP and Hyperparameter Transfer The learning rate that is optimal for a 40-million-parameter proxy model is usually wrong for the 70-billion-parameter model it was meant to stand in for, and muP is the parametrisation designed to make that transfer actually work.