Confidence Intervals and Coverage

An eval run scores 0 correct out of 40. The standard proportion interval, \(\hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/n}\), evaluates to \(0 \pm 0\). The reported 95% confidence interval on the model's accuracy is the single point zero, which is plainly false: 40 failures is entirely compatible with a true accuracy of 5%. The formula did not malfunction; it was applied outside the regime where it means anything.

Coverage is the property that defines an interval. A procedure has 95% coverage if, across repeated experiments, the intervals it produces contain the true parameter 95% of the time. The guarantee attaches to the procedure, not to any particular interval you are holding.

The frequentist claim, stated precisely

Once the data are observed, the interval \([0.31, 0.44]\) either contains the true value or it does not; there is no probability left in it. The 95% describes the long-run behaviour of the recipe that produced it. This is not pedantry, because it explains why you cannot chain intervals, cannot interpret the interval as a distribution over the parameter, and cannot say the true value is "more likely" near the centre without additional assumptions.

The duality with hypothesis testing is exact and more useful than either concept alone: a \((1-\alpha)\) confidence interval is the set of parameter values that would not be rejected at level \(\alpha\). Reading an interval therefore tells you the outcome of every possible test at once, which is why intervals are strictly more informative than the p-value for the single null of zero.

Wald, Wilson, and why the default is bad

For a binomial proportion, the Wald interval inverts a normal approximation centred at \(\hat{p}\) and uses \(\hat{p}\) itself in the variance. Both approximations fail near the boundary, and the failure is severe: actual coverage of the nominal 95% Wald interval oscillates and can fall well below 95% even for moderately large \(n\) (Brown, Cai and DasGupta, 2001, Interval Estimation for a Binomial Proportion, Statistical Science 16(2)).

The Wilson score interval inverts the test using the null variance rather than the estimated one, solving

for \(p\). The result never leaves \([0,1]\), is asymmetric where asymmetry is correct, and gives a sensible answer at \(\hat{p} = 0\): for 0 out of 40 at 95%, roughly \([0, 0.087]\). It costs one quadratic solve. Agresti-Coull's "add two successes and two failures, then apply Wald" is a close approximation that is easier to remember and nearly as good.

For the 0-out-of-\(n\) case specifically there is a memorable shortcut, the rule of three: the upper 95% bound on the rate is approximately \(3/n\). Forty clean runs bound the failure rate at about 7.5%, not at zero.

Confidence versus credible

A credible interval is a statement about the posterior: given the prior and the data, there is 95% posterior probability that the parameter lies in this range. That is the interpretation most readers apply to every interval they see, and it requires a prior.

The two often nearly coincide with weak priors and plenty of data, which is why the confusion survives. They diverge exactly where it matters: small samples, boundary estimates, and hierarchical settings where partial pooling moves the estimate. For 0 successes in 40 trials, a Jeffreys prior gives a credible interval close to the Wilson result, but it is close for a different reason and it licenses the sentence "there is a 95% probability the rate is under 8.7%", which the confidence interval does not.

Neither is more correct. They answer different questions, and the practical guidance is to say which one you computed.

When it breaks

The interval assumes the model. Coverage is guaranteed under the sampling model used to derive it. If observations are correlated (multiple evaluations of the same prompt, multiple events per user), the true variance is larger than the formula assumes and the interval undercovers, often by a lot. Clustered standard errors or a cluster bootstrap fix this; a larger sample does not.

Selection destroys coverage. An interval reported because it excluded zero is conditioned on that event, and conditional coverage is not the nominal coverage. This is why the largest-looking effect in a scan is nearly always smaller on replication, the winner's curse restated in interval form.

Ratios and unbounded quantities. Intervals for ratios can be disconnected or infinite when the denominator's interval spans zero. This is a real feature of the geometry (Fieller's theorem), not a numerical failure, and software that always returns a tidy finite interval for a relative lift is hiding it.

Nominal is not actual. For discrete data, exact intervals such as Clopper-Pearson guarantee at least the nominal coverage, often substantially more, which makes them wider than necessary. Guaranteed conservatism is still a mismatch between the label and the behaviour, and worth stating when precision is being compared across methods.