Pretraining economics, optimisers, RL post-training and scaling behaviour.
The industry's defence against data poisoning was arithmetic: an attacker needs a percentage of the corpus, and a percentage of 260 billion tokens is unobtainable. In October 2025 the largest poisoning study ever run showed the r…
Over 54 days of Llama 3 405B pre-training, the job was interrupted 466 times, roughly once every three hours. At that failure rate the checkpoint interval barely matters; what decides how much of a sixteen-thousand-GPU cluster do…
A 2-billion-parameter model that trades blows with one ten times its size is not an accident of architecture. It is the product of a teacher pouring its full probability distribution into a student, token by token.
Human preference labels are the most expensive ingredient in a modern aligned model. Constitutional AI replaced most of them with a written document and a model judging itself, and the idea quietly took over the alignment stack.
A 75% idle GPU is usually not a compute problem. Across the configurations one VLDB study measured, DNN training spent between 10% and 70% of every epoch blocked on I/O, and the storage system was not busy. The data path has its …
For years the field raced to build bigger models. Then a 2022 study showed most of those giants were undertrained, and quietly redrew the map of how to spend a compute budget.
In 1951 Claude Shannon estimated the entropy of English by having people guess the next letter. In 2023 a 70-billion-parameter language model compressed a gigabyte of Wikipedia to 8.3% of its size, beating every compressor ever p…
An unbiased estimate of every policy's value does not give you an unbiased choice of policy. The moment an optimiser searches over importance-weighted estimates, it goes looking for the estimator's noise, and the history of learn…
Three 7B models that each scored under 30% on Japanese maths were averaged into one that scored 52%. No gradient was computed. Weight-space arithmetic is the cheapest capability gain in the field and the easiest one to fool yours…
For eight years Adam owned every serious pretraining run. Then a competitive-speedrun optimizer that orthogonalizes its own gradient updates scaled from a 124M-parameter toy to a trillion-parameter model with zero loss spikes. He…
A prompt optimiser that never touches a weight has been reported to beat GRPO by around six points using up to 35 times fewer rollouts. That result only makes sense once you stop treating the prompt as writing and start treating …
Replace the reward model with a function that simply checks the answer, and a frontier reasoning model falls out of pure reinforcement learning. The catch is what 'checkable' quietly assumes, and what the model learns to exploit.