Difference-in-Differences and Parallel Trends

A feature launches in three markets and not in five others. Comparing the launched markets before and after is confounded by everything else that changed over that period. Comparing launched to unlaunched markets after the change is confounded by everything that made those markets different. Difference-in-differences takes both differences and subtracts them, so anything constant over time within a market and anything common across markets over time both cancel.

In regression form this is a two-way fixed effects model with unit and time indicators and an interaction term whose coefficient is the effect.

What the assumption actually says

The identifying assumption is parallel trends: absent the treatment, the treated group's average outcome would have followed the same trajectory as the control group's. It says nothing about levels, so the two groups may be arbitrarily different in size, baseline conversion, or composition. It is a statement about the counterfactual path, which is unobserved by construction and therefore not testable.

The standard supporting evidence is pre-trends: if the two groups moved in parallel for several periods before treatment, the assumption is more plausible after. This is evidence, not proof, and it has two known weaknesses. Pre-trend tests are typically underpowered, so failing to reject parallel pre-trends is weak reassurance. And conditioning the analysis on having passed a pre-trend test distorts the subsequent inference, since you selected the sample on a noisy criterion.

Parallel trends is also not invariant to transformation. Trends that are parallel in levels are generally not parallel in logs, so a DiD on raw revenue and a DiD on log revenue are testing different assumptions and can disagree. The choice of scale needs a substantive justification.

The staggered adoption problem

When units adopt treatment at different times, the natural extension is two-way fixed effects with a treatment indicator. This turns out to be badly behaved. The TWFE estimator is a weighted average of many two-by-two comparisons, and some of those comparisons use already-treated units as controls for later-treated ones. When treatment effects change over time, those comparisons receive negative weights, and the overall estimate can have the opposite sign to every individual effect (Goodman-Bacon, 2021, Journal of Econometrics 225(2)).

This is not a small correction. A literature's worth of staggered DiD estimates was revisited on this basis, and the modern estimators, which restrict comparisons to not-yet-treated units and aggregate cohort-by-period effects explicitly, are now the default rather than a robustness check.

When it breaks

Composition changes inside a unit. The estimator assumes the unit is comparable to itself over time. If treatment attracts different users into the treated markets, the post-period population is not the pre-period population and the difference includes a composition shift.

Anticipation invalidates the pre-period. If units change behaviour before treatment begins because they know it is coming, the pre-period is contaminated and the baseline is wrong.

Spillovers to the control group. Contamination of the control units biases the estimate toward zero if the spillover is positive, or away from zero if the control group is harmed. Geographic controls adjacent to treated markets are the common case.

Few treated units make inference unreliable. With three treated markets, cluster-robust standard errors with three clusters are badly anticonservative. Randomisation inference or wild cluster bootstrap are the standard responses, and reporting a conventional t-statistic on a handful of clusters overstates confidence substantially.