Tagged “experimentation”

Reasoning & Evaluation 24 min

Double Machine Learning: How to Use Flexible Models for Causal Estimates Without Inheriting Their Bias

Plug a random forest into a causal regression and the confidence interval tightens around the wrong number, because regularisation bias shrinks more slowly than the standard error. Double machine learning fixes this with two devi…

observational-causal-methods causal-inference confounding estimation ∑ ◫

Platforms & Practice 27 min

Four Ways an A/B Test Lies: Power, Peeking, Sample Ratio Mismatch and Interference

At Microsoft, roughly one third of tested ideas move the metric in the intended direction. That number is only meaningful if the tests are trustworthy, and the four failure modes that destroy trustworthiness all produce a clean, …

experimentation ab-testing causal-inference statistics ∑ ◫

Training & Alignment 24 min

Learning From the Log: Off-Policy Policy Learning, From IPS to Counterfactual Risk Minimisation

An unbiased estimate of every policy's value does not give you an unbiased choice of policy. The moment an optimiser searches over importance-weighted estimates, it goes looking for the estimator's noise, and the history of learn…

policy-learning-and-ope off-policy-evaluation counterfactual importance-sampling ∑ ◫

Reasoning & Evaluation 29 min

The Control Group Is Disappearing: Measuring AI-Assisted Development

Four credible randomised trials of AI coding tools report effects from a 19 percent slowdown to a 56 percent speedup. The newest problem is worse than the spread: developers now refuse to be randomised into the no-AI arm, and 30 …

evaluation measurement experimentation causal-inference ∑ ◫

Platforms & Practice 25 min

The Controls That Make It Worse: Colliders, Mediators and Why Adjusting for Everything Is Wrong

Open surgery beat percutaneous nephrolithotomy on small kidney stones, 93% to 87%. It beat it on large stones too, 73% to 69%. Pooled across both, it lost, 78% to 83%. The arithmetic is correct in all three statements, and no amo…

causal-inference statistics confounding dags ∑ ◫

Reasoning & Evaluation 23 min

The Progress Illusion in Recommender Systems: Weak Baselines, Sampled Metrics and Leaky Splits

In 2019 a careful team could reproduce only 7 of 18 neural recommenders from top venues, and 6 of those 7 lost to nearest-neighbour heuristics. The models were not the problem. The protocol was: untuned baselines, metrics compute…

recommender-systems evaluation metrics benchmarks ∑ ◫

Reasoning & Evaluation 24 min

Twenty Tests, One False Discovery: Multiple Testing From Bonferroni to the False Discovery Rate

A dead Atlantic salmon, scanned in 2009, showed 16 'active' voxels at p below 0.001; every procedure that controlled an error rate across the family found none. This is the argument over what that error rate should be, from Holm …

statistical-inference statistics experimentation ab-testing ∑ ◫

Reasoning & Evaluation 24 min

Your Improvement Is Inside the Noise: Seeds, Nondeterminism and the Reproducibility Problem in ML

Change one bit in one weight of a ResNet and, three epochs later, test accuracy differs by more than ten points. Training is a chaotic process, so a seed is not a control variable but a draw from a distribution. Most published an…

experiment-tracking-and-reproducibility statistics determinism mlops ∑ ◫