Tagged “benchmarks”

Safety, Security & Governance 24 min

A Decade of Adversarial Examples: Why Robustness Never Came Free

In 2014 a perturbation the size of one 8-bit colour step turned a 57.7 percent panda into a 99.3 percent gibbon. Twelve years, 300 million synthetic training images and more than 10^21 training FLOPs later, the best CIFAR-10 mode…

security safety evaluation benchmarks ∑ ◫

Platforms & Practice 24 min

BM25 Refuses to Die: The Probabilistic Relevance Framework Behind Search's Most Durable Baseline

In 2021 a benchmark of 18 retrieval datasets found that a scoring function from 1994 still beat most neural retrievers outside their training domain. BM25 is not a heuristic that got lucky. It is a deliberate approximation of a p…

classical-ir information-retrieval ranking retrieval ∑ ◫

Reasoning & Evaluation 25 min

Error Bars for Evals: Why Most Benchmark Differences Are Noise

A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …

evaluation benchmarks statistics mlops ∑ ◫

Reasoning & Evaluation 24 min

Inside Gradient-Boosted Trees: The Engineering That Made XGBoost, LightGBM and CatBoost Win Tabular ML

XGBoost, LightGBM and CatBoost minimise the same objective with the same kind of tree. What separates them is bookkeeping: XGBoost turned two sums of derivatives into a split score, LightGBM made those sums cheap, and CatBoost ma…

trees-and-ensembles gradient-boosting xgboost tabular ∑ ◫

Reasoning & Evaluation 24 min

The Ladder of Causation: Why No Amount of Observational Data Climbs It Alone

In 2023 GPT-4 scored 97% on a classic cause-and-effect benchmark and 62% on one that hands it the causal graph and asks it to compute. Both results fit a theorem proved in 2020: data from one rung of Pearl's ladder almost never d…

causal-foundations causal-inference counterfactual dags ∑ ◫

Reasoning & Evaluation 27 min

The Last Undefeated Baseline: Why Gradient-Boosted Trees Still Beat Deep Learning on Tabular Data

Deep learning took images in three years and text in five. It has been attacking tabular data since 2016 and still has not won. The reason is not compute or architecture; it is three specific inductive biases that make a tree the…

tabular gradient-boosting deep-learning xgboost ∑ ◫

Reasoning & Evaluation 24 min

The Leaderboard Is Not Your Corpus: Why Top-Ranked Embedding Models Disappoint in Production

Embedding models are chosen from a leaderboard more often than from an experiment, and the leaderboard now publishes training splits for its own test sets. Between contamination, task-family averaging and geometry no benchmark me…

embeddings retrieval-rag evaluation benchmarks ∑ ◫

Reasoning & Evaluation 23 min

The Progress Illusion in Recommender Systems: Weak Baselines, Sampled Metrics and Leaky Splits

In 2019 a careful team could reproduce only 7 of 18 neural recommenders from top venues, and 6 of those 7 lost to nearest-neighbour heuristics. The models were not the problem. The protocol was: untuned baselines, metrics compute…

recommender-systems evaluation metrics benchmarks ∑ ◫

Reasoning & Evaluation 24 min

Twenty Tests, One False Discovery: Multiple Testing From Bonferroni to the False Discovery Rate

A dead Atlantic salmon, scanned in 2009, showed 16 'active' voxels at p below 0.001; every procedure that controlled an error rate across the family found none. This is the argument over what that error rate should be, from Holm …

statistical-inference statistics experimentation ab-testing ∑ ◫

Reasoning & Evaluation 3 min

What the bake-off taught us: classical ML is not dead, it is just under-attended

We pitted twelve sklearn algorithms head-to-head on a tabular dataset. The winner was not the most expensive one. It was not the most modern one. It was the one whose assumptions matched the data.

machine-learning tabular benchmarks evaluation

Reasoning & Evaluation 20 min

When the Judge Is Also a Player: LLM-as-Judge, Contamination, and Why Leaderboards Drift

A strong model grading other models looks like a free lunch for evaluation. It is not. Position, verbosity, and self-preference biases plus quietly leaked test sets mean a leaderboard number can move several points without any mo…

evaluation llm-as-judge benchmarks contamination ∑ ◫

Reasoning & Evaluation 24 min

Your Improvement Is Inside the Noise: Seeds, Nondeterminism and the Reproducibility Problem in ML

Change one bit in one weight of a ResNet and, three epochs later, test accuracy differs by more than ten points. Training is a chaotic process, so a seed is not a control variable but a draw from a distribution. Most published an…

experiment-tracking-and-reproducibility statistics determinism mlops ∑ ◫