Training Dynamics & Scaling
Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.
16concepts
157flashcards
135minutes of reading
- 01 Gradient Clipping A five-character config value, clip norm 1.0, appears in nearly every published pretraining recipe, and it is a crude safety net rather than a fix, one that is easy to misread as more protective than it is.
- 02 Learning-Rate Schedules The shape of the learning-rate curve across a training run matters as much as its peak value, and getting the shape wrong wastes a slice of a compute budget that was never coming back.
- 03 Optimiser State and Memory A model's raw weight size is only the starting point for training memory; Adam alone roughly quadruples the footprint before a single activation has been stored.
- 04 Warmup and Why It Helps The first few hundred to few thousand steps of a large training run are its most fragile, and a short linear ramp on the learning rate is the cheapest insurance against wrecking them.
- 05 Weight Decay in Pretraining Pretraining rarely revisits the same token twice, so the classical overfitting story for weight decay barely applies, yet nearly every LLM recipe still sets it to a nonzero value close to 0.1.