Portfolio ↗

Training Dynamics & Scaling

Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws.

11concepts
53flashcards
94minutes of reading
Start with Gradient Accumulation Drill the deck
All levels beginner intermediate advanced
  1. 01 Gradient Accumulation The trick that lets a single GPU simulate a batch size far larger than what fits in its memory, at the cost of wall-clock time rather than compute. beginner 6m 4 cards