Reasoning & Evaluation

Models that think for longer, and the measurement problems that come with them.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Reasoning & Evaluation 23 min

Clustering Has No Ground Truth: Impossibility, Validation, and What a Cluster Can Promise

In 2002 Jon Kleinberg proved that no clustering function can satisfy three properties almost everyone would ask for. Every algorithm is therefore a definition of what a cluster is, and every validation index is another definition…

clustering unsupervised-learning evaluation metrics ∑ ◫

Reasoning & Evaluation 24 min

Double Machine Learning: How to Use Flexible Models for Causal Estimates Without Inheriting Their Bias

Plug a random forest into a causal regression and the confidence interval tightens around the wrong number, because regularisation bias shrinks more slowly than the standard error. Double machine learning fixes this with two devi…

observational-causal-methods causal-inference confounding estimation ∑ ◫

Reasoning & Evaluation 25 min

Error Bars for Evals: Why Most Benchmark Differences Are Noise

A 250-question benchmark carries a standard error of about three percentage points. Most of the model comparisons published on top of such benchmarks cannot distinguish the models they are comparing. Evaluations are experiments, …

evaluation benchmarks statistics mlops ∑ ◫

Reasoning & Evaluation 23 min

From Features to Circuits: What Attribution Graphs Explain, and the Fraction They Do Not

Swap the Texas features for British Columbia and Claude answers Victoria instead of Austin. That single intervention is the strongest evidence yet that a language model performs genuine multi-step reasoning inside one forward pas…

interpretability mech-interp safety alignment ∑ ◫

Reasoning & Evaluation 23 min

Guarantees Without Calibration: Conformal Prediction and the Limits of LLM Confidence

A language model's stated confidence is a number, not a probability. Conformal prediction offers the opposite trade: it promises nothing about any single answer and something exact about the long run, from any scorer, with one as…

uncertainty evaluation calibration safety ∑ ◫

Reasoning & Evaluation 24 min

Hidden Technical Debt, a Decade On: What Continuous Delivery for ML Actually Fixed

In 2015 a Google paper catalogued the ways machine learning systems rot, and a decade of MLOps tooling set out to pay that debt down. It paid down the debt that lives in pipelines and artefacts, and left the debt that lives in ju…

ci-cd-for-ml mlops evaluation-mlops deployment ∑ ◫

Reasoning & Evaluation 24 min

Inside Gradient-Boosted Trees: The Engineering That Made XGBoost, LightGBM and CatBoost Win Tabular ML

XGBoost, LightGBM and CatBoost minimise the same objective with the same kind of tree. What separates them is bookkeeping: XGBoost turned two sums of derivatives into a split score, LightGBM made those sums cheap, and CatBoost ma…

trees-and-ensembles gradient-boosting xgboost tabular ∑ ◫

Reasoning & Evaluation 23 min

Latent Reasoning: Teaching Language Models to Think Without Tokens

Chain-of-thought made models reason out loud, one word at a time. A new line of work lets them reason in the silent space between words, trading auditability for compute that does not have to be spelled out.

latent-reasoning chain-of-thought test-time-compute reasoning ∑ ◫

Reasoning & Evaluation 18 min

Test-Time Compute: How Reasoning Models Buy Intelligence by the Token

A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.

reasoning test-time-compute inference reinforcement-learning ∑ ◫

Reasoning & Evaluation 24 min

The Bayesian Workflow: Why Fitting a Posterior Is the Easy Part

A 700-draw run of the eight-schools model reported R-hat of 1.01 and put the 2.5% quantile of the between-school spread at 0.89, while the exact posterior holds almost 18% of its mass below that value. Calling a sampler takes one…

bayesian-methods statistics uncertainty calibration ∑ ◫

Reasoning & Evaluation 24 min

The Curse of Dimensionality: Why Distances Stop Meaning Anything, and Why Learning Works Anyway

Scatter 1,000 random points in a 1,000-dimensional cube and the farthest one from a query is only about 12% farther away than the nearest. By that arithmetic nearest-neighbour search should be meaningless, and yet every vector da…

supervised-classical learning-theory embeddings vector-search ∑ ◫

Reasoning & Evaluation 24 min

The Ladder of Causation: Why No Amount of Observational Data Climbs It Alone

In 2023 GPT-4 scored 97% on a classic cause-and-effect benchmark and 62% on one that hands it the causal graph and asks it to compute. Both results fit a theorem proved in 2020: data from one rung of Pearl's ladder almost never d…

causal-foundations causal-inference counterfactual dags ∑ ◫

Browse by topic

Reasoning & Evaluation

Clustering Has No Ground Truth: Impossibility, Validation, and What a Cluster Can Promise

Double Machine Learning: How to Use Flexible Models for Causal Estimates Without Inheriting Their Bias

Error Bars for Evals: Why Most Benchmark Differences Are Noise

From Features to Circuits: What Attribution Graphs Explain, and the Fraction They Do Not

Guarantees Without Calibration: Conformal Prediction and the Limits of LLM Confidence

Hidden Technical Debt, a Decade On: What Continuous Delivery for ML Actually Fixed

Inside Gradient-Boosted Trees: The Engineering That Made XGBoost, LightGBM and CatBoost Win Tabular ML

Latent Reasoning: Teaching Language Models to Think Without Tokens

Test-Time Compute: How Reasoning Models Buy Intelligence by the Token

The Bayesian Workflow: Why Fitting a Posterior Is the Easy Part

The Curse of Dimensionality: Why Distances Stop Meaning Anything, and Why Learning Works Anyway

The Ladder of Causation: Why No Amount of Observational Data Climbs It Alone

Tags