Mediation Analysis: Direct and Indirect Effects

A redesigned onboarding flow raises 90-day retention by 3 points, and it also raises the share of users who connect a data source in week one. Product wants to know how much of the retention gain comes through that connection step, because if most of it does, the cheaper intervention is a prompt to connect a data source. The question sounds like regression bookkeeping. It is one of the hardest identification problems in the causal toolkit.

Three effects that sound alike

Write \(T\) for treatment, \(M\) for the mediator and \(Y(t, m)\) for the outcome under treatment \(t\) with the mediator set to \(m\). \(M(t)\) is the mediator's potential value under \(t\).

The controlled direct effect fixes the mediator for everyone:

It answers a policy question, what does treatment do if we also force \(M = m\), and it can differ for every \(m\).

The natural direct and indirect effects let the mediator take the value it would naturally take (Pearl, 2001, Direct and Indirect Effects, UAI, arXiv:1301.2300):

Their sum is the total effect, \(\mathbb{E}[Y(1, M(1)) - Y(0, M(0))]\), for any outcome model, including nonlinear ones with interactions. That additivity is why natural effects are the quantity most mediation questions actually want.

The cross-world problem

Look at \(Y(1, M(0))\). It is the outcome under treatment, with the mediator at the value it would have taken without treatment. No unit ever experiences both worlds, so no experiment on \(T\), on \(M\), or on both jointly can observe it. Robins and Greenland showed early that direct and indirect effects cannot be separated without special assumptions even when treatment is randomised (Robins and Greenland, 1992, Identifiability and Exchangeability for Direct and Indirect Effects, Epidemiology 3(2)).

Identification of natural effects needs, conditional on pre-treatment covariates \(X\): no unmeasured treatment-outcome confounding, no unmeasured mediator-outcome confounding, no unmeasured treatment-mediator confounding, and a fourth, cross-world condition \(Y(t, m) \perp M(t^*) \mid X\). Under these, the mediation formula gives

The fourth condition fails whenever some variable caused by treatment confounds the mediator and outcome. If the new onboarding also changes how many teammates a user invites, and invites affect both data connection and retention, natural effects are not identified by any data (Avin, Shpitser and Pearl, 2005, Identifiability of Path-Specific Effects, IJCAI).

The responses divide the field. Epidemiologists in the Robins tradition often prefer interventional (randomised) analogues, which replace \(M(0)\) with a random draw from the mediator distribution under control; these are identified with a treatment-induced confounder present, at the price of no longer summing to the total effect in general (VanderWeele, Vansteelandt and Robins, 2014, Epidemiology 25(2)). Structural modellers keep natural effects and argue the cross-world assumption is a reasonable claim about mechanisms.

A worked decomposition

Take a linear model with an interaction and mean-zero noise: \(M = 0.5T + \varepsilon_M\) and \(Y = 1.0\,T + 2.0\,M + 1.5\,T M + \varepsilon_Y\). So \(\mathbb{E}[M(0)] = 0\) and \(\mathbb{E}[M(1)] = 0.5\).

• Total effect: \(\mathbb{E}[Y(1, M(1))] = 1 + 3.5 \times 0.5 = 2.75\); \(\mathbb{E}[Y(0, M(0))] = 0\). TE \(= 2.75\).
• NDE: \(1 + 1.5\,\mathbb{E}[M(0)] = 1.0\).
• NIE: \((2 + 1.5) \times (0.5 - 0) = 1.75\).
• CDE at \(m = 1\): \(1 + 1.5 = 2.5\).

The classic product-of-coefficients recipe (Baron and Kenny, 1986, JPSP 51(6)) multiplies the \(T \to M\) slope by the \(M \to Y\) main effect: \(0.5 \times 2.0 = 1.0\). It misses the 0.75 carried by the interaction and reports the indirect path as smaller than the direct one when the reverse is true. The decomposition is also not unique: evaluating the direct effect at \(M(1)\) instead gives \(1 + 1.5 \times 0.5 = 1.75\) and a pure indirect effect of \(2.0 \times 0.5 = 1.0\). With interaction, which half gets the interaction term is a choice that must be reported.

When it breaks

Randomising treatment does not randomise the mediator. Users who connect a data source differ from those who do not in motivation and team size. Mediator-outcome confounding survives the experiment, and it is the assumption most often violated without comment.

Post-treatment confounders are common in products. Almost any product change alters several behaviours at once. When one of them affects both mediator and outcome, fall back to controlled or interventional effects and say that natural effects are unavailable.

Measurement error in the mediator shifts the split. A noisy proxy for the mediator attenuates the \(M \to Y\) path, moving effect from indirect to direct. "Most of the effect is direct" is often a statement about a bad mediator measurement.

Sensitivity analysis is part of the result. Imai, Keele and Tingley parameterise the correlation between mediator and outcome errors and show how the indirect effect moves as it grows (Imai, Keele and Tingley, 2010, A General Approach to Causal Mediation Analysis, Psychological Methods 15(4)). An indirect effect that vanishes at a small correlation is not a finding to build a roadmap on.