Propensity Scores and Matching

Comparing users who adopted a feature against users who did not is a comparison between two self-selected populations. The obvious fix, matching each adopter to a similar non-adopter, becomes impossible fast: with twenty binary covariates there are over a million cells and most are empty. Rosenbaum and Rubin's result is that you do not need to match on twenty dimensions, only on one (Rosenbaum and Rubin, 1983, The Central Role of the Propensity Score in Observational Studies for Causal Effects, Biometrika 70(1)).

The propensity score is \(e(x) = P(T = 1 \mid X = x)\). Their theorem: if treatment assignment is ignorable given \(X\), it is also ignorable given \(e(X)\) alone. A scalar summary suffices for the whole covariate vector.

Why the balancing property holds

The score is a balancing score: conditional on \(e(X)\), treatment is independent of \(X\). Among units with the same probability of being treated, whether they actually were is as good as random with respect to the covariates. Two units with \(e = 0.3\) may have completely different covariate profiles, and that is fine, because the distribution of \(X\) is the same in both arms at that score.

This is why the checkable output of a propensity analysis is balance, not model fit. A well-fitting propensity model that leaves covariates imbalanced has failed; a poorly calibrated one that achieves balance has succeeded. Standardised mean differences per covariate, before and after adjustment, are the standard report, with values under 0.1 conventionally treated as acceptable. Reporting the propensity model's AUC as evidence of quality inverts the logic: a very high AUC means treated and control units are cleanly separable, which is a positivity problem, not a success.

Four ways to use the score

Matching pairs treated units with control units at similar scores, usually nearest-neighbour within a caliper. It targets the ATT naturally, discards unmatched controls, and makes the comparison population explicit.

Stratification splits the sample into score quintiles and averages within-stratum effects. Simple, and it leaves residual imbalance inside wide strata.

Inverse probability weighting weights treated units by \(1/e(x)\) and controls by \(1/(1-e(x))\), reconstructing a pseudo-population where treatment is independent of covariates. It uses all the data and is extremely sensitive to extreme scores.

Regression on the score is common and the weakest of the four, since it reimposes a functional form on the one quantity the score was supposed to make nonparametric.

When it breaks

It does nothing about unmeasured confounding. This is the point that gets lost, and it is the only one that matters for whether the estimate is causal. The entire method rests on ignorability given the measured \(X\): if selection into treatment depends on motivation, health status, or intent that you did not record, the propensity score balances everything except the thing driving the bias. A perfectly balanced propensity analysis and a badly confounded one look identical in the diagnostics.

Extreme weights destroy precision and stability. A control unit with \(e = 0.02\) receives a weight of 50 and can singlehandedly move the estimate. Trimming units outside a score range and stabilised weights are the standard defences, and both change the estimand: you are now estimating an effect for the subpopulation with overlap, which should be described rather than quietly assumed away.

Positivity failures are structural, not statistical. If no unit with a given profile ever receives treatment, no weighting recovers what would happen if it did. Plotting the score distributions for both arms and looking at the overlap region is the diagnostic, and a large non-overlapping mass means the question cannot be answered for those units at all.

Matching without replacement is order-dependent. Greedy nearest-neighbour matching gives different matched sets depending on the order units are processed. Optimal matching removes this at higher cost, and the difference is rarely reported.

Post-matching standard errors are usually wrong. Treating the matched sample as if it were a simple random sample ignores that matching was estimated from the data. Bootstrap procedures for matching estimators are themselves subtle, and the naive interval is too narrow.