Over 54 days of Llama 3 405B pre-training, the job was interrupted 466 times, roughly once every three hours. At that failure rate the checkpoint interval barely matters; what decides how much of a sixteen-thousand-GPU cluster do…
For eight years Adam owned every serious pretraining run. Then a competitive-speedrun optimizer that orthogonalizes its own gradient updates scaled from a 124M-parameter toy to a trillion-parameter model with zero loss spikes. He…
Replace the reward model with a function that simply checks the answer, and a frontier reasoning model falls out of pure reinforcement learning. The catch is what 'checkable' quietly assumes, and what the model learns to exploit.
GRPO weights every token by its own importance ratio, and on long responses that single-sample estimator quietly poisons the gradient until the model collapses. GSPO moves the ratio up to the whole sequence, and Qwen3's largest m…