Pretraining economics, optimisers, RL post-training and scaling behaviour.
DPO is the right default for almost every preference-tuning project in 2026, but the cases where PPO still wins are sharper and more common than the simplicity pitch admits.
DeepSeek shipped a 1.6T-parameter model in April 2026 with native Ascend inference support while OpenAI committed $500B to Stargate. Both can be right, but only one of them is a moat.
A ten-second utterance is a thousand acoustic frames. Its transcript is forty characters. Nobody wrote down which frames make which character, and the number of ways to line them up is astronomical. Every major speech architectur…
Data-parallel training all-reduces the entire gradient after every step, which is why frontier pretraining happens inside one building with a purpose-built fabric. DiLoCo synchronises every five hundred steps instead of every one…
A network that fits ImageNet with randomly shuffled labels should not generalise on real ones. It does. That single experiment invalidated the textbook account of why machine learning works, and the three phenomena that replaced …
You add a KL penalty to an RLHF objective, log it, and it prints minus 0.03. KL divergence is provably non-negative, and nothing is broken. One formula does four different jobs in modern machine learning, and almost every confusi…
A language model eats trillions of tokens scraped for free. The largest open robot dataset is 527 skills gathered by hand across 21 institutions. That asymmetry, not model capacity, is what makes robot learning hard, and it expla…
Two hundred dollars of API calls pulled more than ten thousand verbatim training examples out of ChatGPT. Memorisation is not a defect that better engineering removes; it scales log-linearly with everything the field is scaling. …
Add 1,000 to every reward in an environment. The optimal policy is unchanged, and the expected policy gradient is unchanged — but the variance of the estimator you actually compute goes up by four orders of magnitude. Every advan…
GRPO weights every token by its own importance ratio, and on long responses that single-sample estimator quietly poisons the gradient until the model collapses. GSPO moves the ratio up to the whole sequence, and Qwen3's largest m…