Training Dynamics & Scaling
Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.
16concepts
157flashcards
135minutes of reading
- 01 Double Descent Test error rises to a peak at the interpolation threshold and then falls again as models grow past it, which breaks the U-shaped bias-variance picture and explains why more parameters can be safe.
- 02 Emergent Abilities and Metric Artefacts Capabilities that appear to switch on abruptly at a critical scale, the argument that the abruptness comes from the metric rather than the model, and what survives the critique.
- 03 Grokking and Delayed Generalisation Networks that memorise a small dataset perfectly and then, thousands of steps later, suddenly generalise, why the delay happens, and what the four competing explanations agree on.
- 04 Loss Spikes and Divergence On almost every long pretraining run the smooth descending loss curve suddenly lurches upward, and diagnosing and recovering from these spikes is still closer to an operational skill than a solved engineering problem.
- 05 Scaling Laws and the Chinchilla Correction How loss falls predictably with compute, parameters, and data, and why the Chinchilla result showed almost every large model of its era was badly undertrained.
- 06 The Critical Batch Size Past a point that can be measured but not derived from first principles, adding more GPUs to widen the batch stops buying faster convergence per token, and just burns compute for redundant gradient information.
- 07 The Edge of Stability Gradient descent on neural networks does not stay in the regime where its convergence theory applies; the loss curvature grows until training is marginally unstable, and then hovers there.
- 08 The Neural Tangent Kernel and Feature Learning In the infinite-width limit a network trains like a fixed kernel machine, which makes it analysable and also strictly weaker than real networks, and the gap between the two regimes is where deep learning's advantage lives.
- 09 Warmup-Stable-Decay Schedules Cosine decay bakes the total step count into its formula before training starts; warmup-stable-decay schedules split the learning-rate curve so that commitment can be made at the very end instead.
- 10 muP and Hyperparameter Transfer The learning rate that is optimal for a 40-million-parameter proxy model is usually wrong for the 70-billion-parameter model it was meant to stand in for, and muP is the parametrisation designed to make that transfer actually work.