Tagged “metrics”

5 posts.

All Model Architecture23 Training & Alignment22 Inference & Serving24 Agents & Orchestration13 Reasoning & Evaluation28 Safety, Security & Governance9 Platforms & Practice22

Reasoning & Evaluation 23 min

Clustering Has No Ground Truth: Impossibility, Validation, and What a Cluster Can Promise

In 2002 Jon Kleinberg proved that no clustering function can satisfy three properties almost everyone would ask for. Every algorithm is therefore a definition of what a cluster is, and every validation index is another definition…

clustering unsupervised-learning evaluation metrics ∑ ◫

Inference & Serving 24 min

Designing for a Collaborator That Is Sometimes Wrong: The Evidence Behind Human-AI Interaction Design

In a 2025 randomized trial, experienced developers using AI tools took 19% longer to finish their tasks while believing they had been 20% faster. Twenty-six years of human-AI interaction research explain the gap: an assistant's v…

interaction-design-for-ai product verification calibration ∑ ◫

Platforms & Practice 27 min

Four Ways an A/B Test Lies: Power, Peeking, Sample Ratio Mismatch and Interference

At Microsoft, roughly one third of tested ideas move the metric in the intended direction. That number is only meaningful if the tests are trustworthy, and the four failure modes that destroy trustworthiness all produce a clean, …

experimentation ab-testing causal-inference statistics ∑ ◫

Reasoning & Evaluation 23 min

The Progress Illusion in Recommender Systems: Weak Baselines, Sampled Metrics and Leaky Splits

In 2019 a careful team could reproduce only 7 of 18 neural recommenders from top venues, and 6 of those 7 lost to nearest-neighbour heuristics. The models were not the problem. The protocol was: untuned baselines, metrics compute…

recommender-systems evaluation metrics benchmarks ∑ ◫

Safety, Security & Governance 24 min

You Cannot Have All Three: COMPAS, Calibration and the Impossibility Theorems of Fair Classification

In 2016 ProPublica showed that COMPAS wrongly flagged 44.9% of Black defendants who never reoffended against 23.5% of white ones, and its vendor showed the scores meant the same thing for both groups. Both were right, and a one-l…

fairness-and-bias fairness bias calibration ∑ ◫