Estimators, Bias, Variance and Consistency

Two teams estimate the same click-through rate from the same 200 impressions. One reports the raw sample proportion, 3/200 = 1.5%. The other reports a shrunk estimate, (3 + 2)/(200 + 40) = 2.1%, pulling toward a 5% prior taken from similar surfaces. The first estimate is unbiased. The second is not. The second is closer to the truth on almost every replication of this experiment, and that is the whole argument for why unbiasedness is a property, not a goal.

An estimator is a function of data, \(\hat{\theta} = g(X_1, \dots, X_n)\). Because the data are random, the estimator is random. Everything worth saying about it is a statement about that distribution.

The risk decomposition

The standard way to score an estimator is mean squared error, and it splits exactly:

The derivation is one line of adding and subtracting \(\mathbb{E}[\hat{\theta}]\), and the cross term vanishes because \(\mathbb{E}[\hat{\theta} - \mathbb{E}[\hat{\theta}]] = 0\). Bias is systematic error: where the estimator lands on average. Variance is instability: how far it moves when you resample.

This is the same decomposition that governs model complexity in supervised learning, and it is worth noticing that it is a statement about a fixed estimator across hypothetical repeated samples, not about a fixed dataset. The reason regularisation works, the reason shrinkage works, and the reason the 2.1% estimate above beats the 1.5% one all reduce to the same trade: accept a little bias to remove a lot of variance.

Stein's paradox is the sharp version. For estimating three or more independent normal means simultaneously under total squared error, the obvious unbiased estimator (each sample mean) is inadmissible: the James-Stein estimator, which shrinks all of them toward a common point, has strictly lower risk for every value of the true means. Shrinkage is not a heuristic; in dimension three and above it is a theorem.

Consistency and efficiency

Consistency is an asymptotic guarantee: \(\hat{\theta}_n \to \theta\) in probability as \(n \to \infty\). It is a weak requirement and almost every estimator worth using has it. An estimator can be biased at every finite \(n\) and still be consistent, which is exactly the situation for maximum likelihood in most models. The sample variance with the \(1/n\) divisor is biased downward for every \(n\), and consistent.

Efficiency asks how small the variance can get. For an unbiased estimator of a scalar parameter, the Cramér-Rao bound puts a floor at the inverse Fisher information:

Fisher information measures how sharply the log-likelihood curves at the truth. A flat likelihood means the data barely distinguish nearby parameter values, so no estimator can be precise. Maximum likelihood attains this bound asymptotically, which is the technical content of the claim that MLE is the default choice.

Note the qualifier: the bound applies to unbiased estimators. Biased estimators routinely beat it, which is not a contradiction; it is the reason the bound is rarely the last word.

Plug-in estimation and the delta method

Most quantities you care about are functions of simpler ones. If \(\hat{\theta}\) is approximately normal around \(\theta\) with standard error \(\sigma / \sqrt{n}\), then for a differentiable \(g\),

so \(g(\hat{\theta})\) is approximately normal with standard error \(|g'(\theta)| \sigma / \sqrt{n}\). That is the delta method, and it is how a standard error on a conversion rate becomes a standard error on a log-odds or a ratio of rates without any new theory.

It also warns you where things break. When \(g'(\theta) = 0\) the first-order term vanishes and the normal approximation fails; the estimator's distribution becomes chi-squared-like rather than symmetric. Ratios whose denominator can approach zero are the common practical case, and they are why relative-lift confidence intervals get wildly asymmetric on low-traffic segments.

When it breaks

Asymptotics arrive late in the tail. Normal approximations for a proportion are reasonable near \(p = 0.5\) and poor near the boundary. With \(p = 0.002\) and \(n = 1000\), the expected count is 2 and the Wald interval can include negative values. The fix is an exact or score-based interval, not a bigger sample you do not have.

Model misspecification silently rewrites the target. Under misspecification, maximum likelihood still converges, but to the parameter that minimises KL divergence to the true distribution rather than to any true parameter. The estimator is consistent for the wrong thing, and the reported standard errors are wrong too unless you use a sandwich variance.

Independence is usually assumed and rarely true. Every \(\sqrt{n}\) in this note assumes independent observations. Clustered data (multiple events per user, repeated measurements per device) inflate true variance while the naive formula keeps shrinking, which produces confident nonsense. The effective sample size is the number of independent units, and for most product metrics that is users, not events.