Heterogeneous Treatment Effects and Uplift

A retention offer lifts renewal by 2 percentage points on average. That number decides whether to run the campaign. It does not decide who to send it to, and those are different problems with different answers: the offer may lift renewal by 9 points among users who were wavering, do nothing for users who would have renewed regardless, and actively annoy a segment into churning.

The target is the conditional average treatment effect, \(\tau(x) = \mathbb{E}[Y(1) - Y(0) \mid X = x]\). The obstacle is structural rather than statistical: \(\tau(x)\) is never observed for any unit, because each unit reveals one potential outcome. Supervised learning has labels; CATE estimation does not.

Why "predict who responds" is the wrong model

The instinct is to train a model on treated users predicting the outcome, and target the users it scores highly. That model estimates \(\mathbb{E}[Y \mid X, T=1]\), which ranks users by likelihood of the outcome, not by effect of treatment. The two orderings differ exactly where it matters: a user certain to renew scores at the top and has zero uplift.

The four-way segmentation makes this concrete. Persuadables convert only if treated, and they are the entire value of the campaign. Sure things convert either way, so treating them is wasted cost. Lost causes convert neither way, also wasted. Sleeping dogs convert only if left alone, so treating them destroys value. A response model targets sure things and persuadables indiscriminately; an uplift model separates them.

The meta-learners

Künzel and colleagues organised the common approaches as meta-algorithms that wrap any supervised learner (Künzel, Sekhon, Bickel and Yu, 2019, PNAS 116(10), 4156-4165).

S-learner fits one model on all data with treatment as a feature, and takes \(\hat{\tau}(x) = \hat{\mu}(x, 1) - \hat{\mu}(x, 0)\). Simple, and it biases toward zero: a regularised learner facing one weakly predictive binary feature among hundreds may effectively drop it, producing \(\hat{\tau} \approx 0\) everywhere.

T-learner fits separate models on treated and control groups and differences them. It cannot ignore treatment, and it inherits both models' errors: when one arm is much smaller, its noisier fit dominates the difference, and the two models' independent regularisation choices leave structure in \(\hat{\tau}\) that is an artefact of fitting, not a real effect.

X-learner imputes each unit's individual effect using the other arm's model, then regresses those imputed effects on covariates, weighting the two resulting estimates by the propensity score. It is designed for the unbalanced case, common in practice where treated groups are small, and is the strongest of the three when group sizes differ substantially.

Causal forests take a different route, adapting random forest splitting to maximise heterogeneity in treatment effect rather than in outcome, with honest sample splitting so the same data do not choose the split and estimate the effect. This yields asymptotically normal estimates with valid confidence intervals (Wager and Athey, 2018, JASA 113(523)).

Evaluating without ground truth

Standard model selection is unavailable, since there is no held-out \(\tau\) to score against. The practical instruments rank rather than score.

The Qini and uplift curves sort the held-out population by predicted uplift, then plot the cumulative incremental outcome as more of the population is treated. A model with real signal shows a steep early rise and a peak before 100%; a useless model traces the diagonal. The area between the curve and the diagonal is the summary statistic.

Treatment-effect deciles are the blunter and often more convincing check: bin the held-out set by predicted uplift and compute the actual randomised difference within each bin. If the top decile shows a large measured effect and the bottom shows none or a negative one, the model has found real heterogeneity. This works only on experimental data, which is why uplift modelling nearly always requires a randomised training set.

When it breaks

Heterogeneity is a weaker signal than the main effect, so it needs far more data. Detecting a 2-point average effect might take 30,000 users; reliably distinguishing a 9-point segment from a 0-point one takes considerably more, since the estimand is a difference of differences. Most uplift models fitted on experiments powered for the ATE are fitting noise.

Noise looks exactly like heterogeneity. Any flexible model will produce varying \(\hat{\tau}(x)\) on random data, and the resulting deciles will look ordered because they were sorted by the model's own output. Only an out-of-sample decile plot on randomised data distinguishes signal from sorting.

Effect heterogeneity is scale-dependent. A treatment with a constant effect on the log-odds has a varying effect on the probability, peaking at 0.5. Discovering "heterogeneity" that is an artefact of the link function is common, and checking whether it persists on the natural scale is a one-line test.

Targeting on estimated uplift creates a feedback loop. Once the policy targets the model's top segment, future logged data has little variation in the bottom segments, so the model cannot be reliably refit. Maintaining a small randomised holdout permanently is the standard defence and should be budgeted as an ongoing cost.