Model Architecture
33 min
The Training Run That Never Ends: Continued Pretraining, Mid-Training and Model Growth
Forty million tokens, a rounding error against Llama 3's fifteen trillion, moved GSM8K by 24 percent. That is not a pretraining result; it is a result about what happens after pretraining is nominally over. A pretrained checkpoin…