Tagged “calibration”

4 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving20 Agents & Orchestration11 Reasoning & Evaluation27 Safety, Security & Governance7 Platforms & Practice22

Inference & Serving 24 min

Designing for a Collaborator That Is Sometimes Wrong: The Evidence Behind Human-AI Interaction Design

In a 2025 randomized trial, experienced developers using AI tools took 19% longer to finish their tasks while believing they had been 20% faster. Twenty-six years of human-AI interaction research explain the gap: an assistant's v…

interaction-design-for-ai product verification calibration ∑ ◫

Reasoning & Evaluation 23 min

Guarantees Without Calibration: Conformal Prediction and the Limits of LLM Confidence

A language model's stated confidence is a number, not a probability. Conformal prediction offers the opposite trade: it promises nothing about any single answer and something exact about the long run, from any scorer, with one as…

uncertainty evaluation calibration safety ∑ ◫

Reasoning & Evaluation 24 min

The Bayesian Workflow: Why Fitting a Posterior Is the Easy Part

A 700-draw run of the eight-schools model reported R-hat of 1.01 and put the 2.5% quantile of the between-school spread at 0.89, while the exact posterior holds almost 18% of its mass below that value. Calling a sampler takes one…

bayesian-methods statistics uncertainty calibration ∑ ◫

Safety, Security & Governance 24 min

You Cannot Have All Three: COMPAS, Calibration and the Impossibility Theorems of Fair Classification

In 2016 ProPublica showed that COMPAS wrongly flagged 44.9% of Black defendants who never reoffended against 23.5% of white ones, and its vendor showed the scores meant the same thing for both groups. Both were right, and a one-l…

fairness-and-bias fairness bias calibration ∑ ◫