A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …
Chain-of-thought made models reason out loud, one word at a time. A new line of work lets them reason in the silent space between words, trading auditability for compute that does not have to be spelled out.
Replace the reward model with a function that simply checks the answer, and a frontier reasoning model falls out of pure reinforcement learning. The catch is what 'checkable' quietly assumes, and what the model learns to exploit.
Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…
A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.
o3, R1 and Claude extended thinking are a real capability shift on a narrow slice of tasks. They are also being shoved into product surfaces that punish every property reasoning models exhibit - and the bill is starting to arrive.