Sampling Distributions and the Central Limit Theorem

A checkout experiment reports average order value of $41.20 from 30 orders, with a standard error of $3.10. Nobody ever saw a distribution with spread $3.10. The orders themselves might range from $4 to $900. The standard error describes a hypothetical object: the spread of the average you would get if you could rerun the same 30-order sample thousands of times. That object is the sampling distribution, and nearly every interval and test in applied statistics is a claim about its shape.

The object that inference is about

Let \(X_1, \dots, X_n\) be independent draws from a distribution with mean \(\mu\) and variance \(\sigma^2\). Any statistic \(T_n = t(X_1,\dots,X_n)\) is itself a random variable, and its distribution across hypothetical repetitions of the sampling is its sampling distribution. For the sample mean \(\bar X_n\), two facts hold exactly at every \(n\) with no normality assumption:

The standard error \(\sigma/\sqrt{n}\) is the standard deviation of that sampling distribution, and it shrinks at the square-root rate: four times the data halves the uncertainty. What the two formulas do not give you is the shape, and the shape is what turns a standard error into a 95% interval.

The distinction between the population distribution, the sample's empirical distribution and the sampling distribution of a statistic is the most commonly blurred one in the subject. A histogram of your 30 orders estimates the first. The second is what the bootstrap resamples. The third is what a confidence interval needs, and it is never directly observed.

What the theorem promises

The classical (Lindeberg–Lévy) central limit theorem says that if the \(X_i\) are i.i.d. with finite variance,

meaning the standardised mean's distribution function converges pointwise to the standard normal \(\Phi\). It is a limit statement. It says nothing about any particular \(n\), which is the only thing a practitioner has.

The Berry–Esseen theorem supplies the missing rate. With \(\rho = \mathbb{E}|X - \mu|^3\) finite,

(Berry, 1941, The Accuracy of the Gaussian Approximation to the Sum of Independent Variates, Trans. AMS 49(1)). The constant has been pushed down over decades; for identically distributed summands \(C \le 0.4748\) (Shevtsova, 2011, arXiv:1111.6554). The error shrinks like \(1/\sqrt{n}\) and grows with the standardised third moment, which is a measure of how lopsided the population is.

A worked case: the exponential

Take \(X \sim \text{Exp}(1)\), a reasonable toy for waiting times or order values. Here \(\mu = \sigma = 1\) and \(\rho = 12/e - 2 \approx 2.41\). At \(n = 30\) the Berry–Esseen bound is \(0.4748 \times 2.41 / \sqrt{30} \approx 0.21\), which sounds alarming. The exact sampling distribution is available, since \(n\bar X_n\) is Gamma\((n,1)\), and the actual worst-case gap to the normal is about \(0.024\). The bound is a worst case over all distributions with that third moment, and it is loose by nearly an order of magnitude here.

The gap that does matter shows up in the tails. A nominal two-sided 95% interval \(\bar X \pm 1.96/\sqrt{30}\) (treating \(\sigma\) as known) misses high about 3.4% of the time and low about 1.4%. Total coverage looks fine at roughly 95.2%, but the errors are lopsided by a factor of more than two. At \(n = 100\) the split is still roughly 3.0% versus 1.9%. The mean of a right-skewed population has skewness \(\gamma/\sqrt{n}\), where \(\gamma\) is the population skewness, so the asymmetry decays slowly, and one-sided decisions ("is revenue per user above target?") inherit it in full.

The "\(n \ge 30\) is enough" rule of thumb is folklore rather than a theorem, and practitioners disagree about whether it should be taught at all. For the exponential, with skewness 2, it is roughly adequate for two-sided intervals. For revenue data with skewness of 20 or 50, which is common when a few whale customers dominate, the same logic suggests you need hundreds of times more data before the skew of the mean falls to the exponential's level at \(n = 30\).

When it breaks

Infinite variance ends the story. If \(\sigma^2 = \infty\) the classical theorem does not apply. The mean of \(n\) standard Cauchy draws is exactly standard Cauchy, no more concentrated than one draw. Pareto-tailed quantities such as file sizes, claim amounts and some revenue distributions sit close to this regime, and the sample variance then keeps jumping as new extreme values arrive.

Dependence shrinks the effective sample. For a stationary series with lag-\(k\) autocorrelation \(\rho_k\), the variance of the mean is approximately \(\frac{\sigma^2}{n}\left(1 + 2\sum_{k\ge1}\rho_k\right)\). With \(\rho_1 = 0.5\) decaying geometrically, the bracket equals 3, so 3,000 correlated observations carry about the information of 1,000 independent ones. Treating sessions from the same user as independent is the everyday version of this mistake.

Plugging in \(\hat\sigma\) adds a second approximation. Replacing \(\sigma\) with the sample standard deviation gives the \(t\) statistic, and under skew \(\bar X\) and \(s\) are correlated: samples that happen to include a large value have both a high mean and a high spread. That correlation distorts the \(t\) statistic's tails beyond what the CLT for \(\bar X\) alone predicts.

Not every statistic is a mean. The CLT covers smooth functions of means through the delta method, but the sample maximum, extreme quantiles, and statistics at a parameter boundary converge to non-normal limits or at different rates. For those, a normal interval is not approximately right, it is the wrong family.