In 2021 a benchmark of 18 retrieval datasets found that a scoring function from 1994 still beat most neural retrievers outside their training domain. BM25 is not a heuristic that got lucky. It is a deliberate approximation of a p…
In 2002 Jon Kleinberg proved that no clustering function can satisfy three properties almost everyone would ask for. Every algorithm is therefore a definition of what a cluster is, and every validation index is another definition…
Plug a random forest into a causal regression and the confidence interval tightens around the wrong number, because regularisation bias shrinks more slowly than the standard error. Double machine learning fixes this with two devi…
A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …
At Microsoft, roughly one third of tested ideas move the metric in the intended direction. That number is only meaningful if the tests are trustworthy, and the four failure modes that destroy trustworthiness all produce a clean, …
A language model's stated confidence is a number, not a probability. Conformal prediction offers the opposite trade: it promises nothing about any single answer and something exact about the long run, from any scorer, with one as…
A 700-draw run of the eight-schools model reported R-hat of 1.01 and put the 2.5% quantile of the between-school spread at 0.89, while the exact posterior holds almost 18% of its mass below that value. Calling a sampler takes one…
Open surgery beat percutaneous nephrolithotomy on small kidney stones, 93% to 87%. It beat it on large stones too, 73% to 69%. Pooled across both, it lost, 78% to 83%. The arithmetic is correct in all three statements, and no amo…
In the fall of 1960 Rudolf Kalman presented a paper at NASA Ames that engineers found hard to grasp; by early 1961 it was navigating simulated spacecraft around the Moon. The same recursion now computes the exact likelihood of ev…
In 2023 GPT-4 scored 97% on a classic cause-and-effect benchmark and 62% on one that hands it the causal graph and asks it to compute. Both results fit a theorem proved in 2020: data from one rung of Pearl's ladder almost never d…
In 2019 a careful team could reproduce only 7 of 18 neural recommenders from top venues, and 6 of those 7 lost to nearest-neighbour heuristics. The models were not the problem. The protocol was: untuned baselines, metrics compute…
For decades the normalising constant made energy-based models nearly impossible to train at scale. Between 2019 and 2022 the field stopped computing it and learned its gradient instead, and score matching, Langevin sampling and d…