Potential Outcomes and the Fundamental Problem

Users who saw the new onboarding flow retained at 62%; users who did not retained at 48%. The fourteen-point gap is a comparison between two different groups of people, and it answers the causal question only if those groups were otherwise identical. If the new flow shipped to users on the latest app version, it is a statement about people who update their apps.

The potential outcomes framework makes this precise (Rubin, 1974, Journal of Educational Psychology 66(5)). Each unit \(i\) has two potential outcomes: \(Y_i(1)\), what would happen under treatment, and \(Y_i(0)\), what would happen without it. The individual causal effect is \(\tau_i = Y_i(1) - Y_i(0)\).

The problem that has no solution

For any unit you observe exactly one of the two. The other is counterfactual and is not merely unmeasured, it is undefined in the world that occurred. Holland named this the fundamental problem of causal inference, and its consequence is that no individual causal effect is ever identified from data (Holland, 1986, Statistics and Causal Inference, JASA 81(396)).

What is recoverable is an average. The average treatment effect is \(\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)]\), and because expectation is linear this equals \(\mathbb{E}[Y(1)] - \mathbb{E}[Y(0)]\), two population quantities that can be estimated from different sets of units. Causal inference is therefore a missing-data problem: half the table is unobserved, and every method is an argument about how to fill it.

Two other estimands matter and are frequently confused with the ATE. The ATT conditions on the treated, \(\mathbb{E}[Y(1) - Y(0) \mid T = 1]\), which is what you want when asking whether a programme helped the people who enrolled. The CATE conditions on covariates, \(\mathbb{E}[Y(1)-Y(0) \mid X = x]\), which is what personalisation and targeting need. When effects are heterogeneous, these differ, and a paper reporting "the effect" without saying which one is ambiguous.

Why randomisation works

The naive comparison estimates \(\mathbb{E}[Y \mid T=1] - \mathbb{E}[Y \mid T=0]\), which equals \(\mathbb{E}[Y(1)\mid T=1] - \mathbb{E}[Y(0)\mid T=0]\). That is the ATE only if the treated group's outcome under treatment represents everyone's, and likewise for control. Randomisation makes treatment assignment independent of the potential outcomes,

so conditioning on \(T\) tells you nothing about which units you selected, and the two conditional expectations become the unconditional ones. This is why an experiment identifies the ATE without any model of the outcome: the design, not the analysis, does the work.

Balance on observed covariates is a consequence of randomisation, not the mechanism. The reason randomisation is uniquely strong is that it balances unobserved covariates too, in expectation, and no observational method can claim that.

SUTVA, the assumption nobody states

Writing \(Y_i(t)\) with a single index assumes two things. First, no interference: unit \(i\)'s outcome depends only on unit \(i\)'s treatment, not on anyone else's. Second, no hidden versions of treatment: "treated" means one well-defined thing.

Both fail routinely. In a marketplace, treating some sellers changes the outcomes of untreated sellers competing for the same buyers. In a social product, a treated user's behaviour changes their friends'. In each case the potential outcome notation is not just imprecise, it is undefined, because unit \(i\) has as many potential outcomes as there are assignment vectors for everyone else.

When it breaks

Heterogeneity makes "the effect" the wrong noun. An ATE of zero is compatible with a large positive effect on half the population and an equal negative effect on the other half. Reporting a single average as the finding, when the design has power to look at subgroups, hides the actionable result.

External validity is a separate argument. Randomisation identifies the effect in the population sampled, at the time sampled, under the specific version of the treatment deployed. It says nothing about a different population or a different implementation, and the leap from one to the other is a domain argument that no statistical procedure supplies.

Non-compliance breaks the mapping from assignment to treatment. If assigned users do not all receive treatment, the randomised comparison estimates the effect of assignment, the intention-to-treat effect, not of treatment. That is often the decision-relevant quantity anyway; treating it as the treatment effect overstates the effect on those who complied.

Conditioning on a post-treatment variable destroys the guarantee. Randomisation makes \(T\) independent of potential outcomes at the moment of assignment. Restricting the analysis to users who completed onboarding, an event affected by treatment, reintroduces selection and the comparison is no longer randomised.