Models that think for longer, and the measurement problems that come with them.
A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …
Chain-of-thought made models reason out loud, one word at a time. A new line of work lets them reason in the silent space between words, trading auditability for compute that does not have to be spelled out.
A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.
o3, R1 and Claude extended thinking are a real capability shift on a narrow slice of tasks. They are also being shoved into product surfaces that punish every property reasoning models exhibit - and the bill is starting to arrive.
We pitted twelve sklearn algorithms head-to-head on a tabular dataset. The winner was not the most expensive one. It was not the most modern one. It was the one whose assumptions matched the data.
A strong model grading other models looks like a free lunch for evaluation. It is not. Position, verbosity, and self-preference biases plus quietly leaked test sets mean a leaderboard number can move several points without any mo…
The model layer is commoditising and the answer flips every six months. The only durable advantage is the ability to A/B a model swap end-to-end in 48 hours and know whether it improved things for your users.