Platforms & Practice

The Controls That Make It Worse: Colliders, Mediators and Why Adjusting for Everything Is Wrong

Open surgery beat percutaneous nephrolithotomy on small kidney stones, 93% to 87%. It beat it on large stones too, 73% to 69%. Pooled across both, it lost, 78% to 83%. The arithmetic is correct in all three statements, and no amount of data tells you which one answers the clinical question.

In 1986, a study of 700 kidney stone patients at a London hospital compared open surgery against percutaneous nephrolithotomy. The results, which have been taught ever since, look like this:

Stone size Open surgery Percutaneous
Small 93% (81/87) 87% (234/270)
Large 73% (192/263) 69% (55/80)
Combined 78% (273/350) 83% (289/350)

Open surgery wins in both subgroups and loses overall (Charig et al., 1986, British Medical Journal 292). Nothing is miscalculated. The reversal happens because surgeons sent the hard cases, the large stones, to open surgery, so its pooled number carries the difficulty of its caseload.

Here is the part that matters for anyone building models rather than treating patients: the data cannot tell you which number is the answer. Both are valid summaries of the same table. Deciding which one estimates the causal effect requires knowing what role stone size plays, and that knowledge does not come from the numbers. It comes from knowing what causes what.

That single observation, generalised, dismantles the most widespread heuristic in applied statistics: control for everything you can measure. Some variables must be adjusted for. Some must not be, because adjusting for them creates bias that was not there. And a variable can be in either category with the same name, the same distribution and the same correlations, depending only on the causal structure.

Why this matters: Every regression, every propensity model, every "we controlled for confounders" sentence in a model card involves choosing an adjustment set. If that set includes a collider or a mediator, the resulting estimate is biased in a direction you cannot determine from the output, and the confidence interval will be tight and wrong. This is not an obscure statistical corner; it is the default outcome of throwing all available columns into a model and reading the coefficients.

TL;DR

  • Adjustment is not monotone in safety. Adding a variable to the control set can introduce bias rather than reduce it, and the standard heuristic of controlling for everything measurable reliably produces this.
  • Three junction types exist and they behave in opposite ways: a fork (confounder) is open and closes on conditioning; a chain (mediator) is open and closes on conditioning; a collider is closed by default and opens on conditioning.
  • Conditioning on a collider is what happens whenever you restrict analysis to a selected population: hospitalised patients, converted users, accepted applicants, or anyone who reached step three of the funnel.
  • Collider bias is not theoretical. Early COVID-19 risk factor studies drawn from tested or hospitalised samples were shown to carry associations manufactured entirely by the sampling.
  • The backdoor criterion turns "which variables should I control for" into a mechanical check against a graph, and the graph is an assumption you assert, not something the data reveals.
  • Four common adjustments are actively harmful: a mediator when the total effect is wanted, a descendant of the outcome, a pure instrument, and any post-treatment variable in a randomised experiment.
  • Adjusting for a pure cause of treatment does not reduce bias; it amplifies whatever bias remains from unmeasured confounding, while also inflating variance.
  • Identification is logically prior to estimation. If the effect is not identified, more data buys a tighter interval around the wrong quantity.

At a Glance

flowchart LR
    Q["Question:<br/>effect of X on Y"] --> G["Draw the graph,<br/>including unmeasured causes"]
    G --> B{"Backdoor criterion:<br/>block all paths into X<br/>without touching<br/>descendants of X"}
    B -->|"a valid set exists"| ID["Identified"]
    B -->|"none exists"| ALT["Front door? IV?<br/>Or say it cannot<br/>be answered"]
    ID --> OV{"Overlap:<br/>every covariate value<br/>gets both treatments?"}
    OV -->|"yes"| EST["Estimate:<br/>regression, weighting, DML"]
    OV -->|"no"| RESTRICT["Restrict to the<br/>overlap subpopulation<br/>and say so"]
    EST --> SENS["Sensitivity analysis:<br/>how strong must an<br/>unmeasured confounder be?"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class Q,G blue
    class B,OV amber
    class ID,EST teal
    class ALT,RESTRICT rose
    class SENS purple

Note the order. Identification comes before estimation, and both come before any model is fitted. Reversing that order is how a well-specified regression ends up estimating something nobody wanted.

How the Field Learned That More Control Is Not Better

The reversal in the kidney stone table was described by Yule in 1903 and formalised by Simpson in 1951, so the phenomenon has been known for over a century. What took much longer was the machinery to say which aggregation is correct.

The intervening decades produced the pieces separately. Berkson noticed in 1946 that hospital-based studies manufacture associations between diseases: if either of two conditions raises the probability of admission, then among admitted patients they appear negatively correlated even when independent in the population. Rubin's potential outcomes framework in the 1970s made "causal effect" a precise quantity, a contrast between two outcomes for the same unit, and made explicit that only one is ever observed. Pearl's graphical framework in the 1980s and 1990s supplied the missing piece: a language in which "which variables should I adjust for" has a mechanical answer given a stated structure.

timeline
    title From Paradox to Procedure
    1903 : Yule describes association reversal under aggregation
    1946 : Berkson shows hospital-based samples manufacture disease associations
    1951 : Simpson formalises the reversal that now carries his name
    1974 : Rubin defines causal effects via potential outcomes; only one is ever observed
    1986 : Holland names the fundamental problem of causal inference
         : Charig et al. publish the kidney stone data that became the canonical example
    1995 : Pearl introduces the do-calculus and the backdoor criterion
    2020 : Griffith et al. show collider bias distorting COVID-19 risk factor studies from selected samples
    2024 : Cinelli, Forney and Pearl publish a taxonomy of good and bad controls

The 2024 taxonomy is worth singling out, because it exists for a reason that says something about the field (Cinelli, Forney and Pearl, 2024, A Crash Course in Good and Bad Controls, Sociological Methods & Research 53(3), 1071-1104). Nearly thirty years after the backdoor criterion, a paper enumerating which controls help and which hurt was still needed, because the "adjust for everything" heuristic had not budged.

[IMAGE: Three-panel figure showing the same 2x2 contingency table three ways. Panel 1: raw pooled counts with treatment B ahead. Panel 2: the same data split by stone size with treatment A ahead in both strata, and a callout showing the caseload imbalance (263 large stones to A, 270 small to B). Panel 3: a DAG with stone size pointing to both treatment choice and outcome, annotated "this is why the disaggregated table is the causal one". Caption: "The third panel is the only one that contains the answer, and it is not made of data."]

The Three Junctions

Every path between two variables in a graph is built from three structures. They behave differently under conditioning, and the collider behaves in the opposite direction to the intuition most people bring.

Fork: \(X \leftarrow C \to Y\)

\(C\) is a common cause. The path is open, so \(X\) and \(Y\) are associated even with no causal link between them, and conditioning on \(C\) closes it. This is confounding, and it is the case the "control for everything" heuristic has in mind. Stone size in the kidney data is a fork: it affects which treatment the surgeon chooses and it affects whether the patient recovers.

Chain: \(X \to M \to Y\)

\(M\) is a mediator. The path is open, and conditioning on \(M\) closes it. Here closing the path is usually wrong. If you want the total effect of \(X\) on \(Y\) and \(X\) works through \(M\), adjusting for \(M\) removes precisely the effect you were measuring. A drug that lowers blood pressure, analysed within blood-pressure strata, appears to do nothing.

[IMAGE: Scatter plot of talent against luck for a simulated population, showing zero correlation across the full cloud of grey points. The subset that was hired is highlighted and occupies the upper-right region above a diagonal admission boundary; a regression line fitted to the highlighted subset alone slopes clearly downward. Caption: "Two independent variables, one selection rule, and a negative correlation that exists only in the sample."]

Collider: \(X \to C \leftarrow Y\)

\(C\) is a common effect. The path is closed by default, so \(X\) and \(Y\) are unassociated through it. Conditioning on \(C\), or on any descendant of \(C\), opens it and creates an association that does not exist in the population.

The mechanism is simple once seen. Suppose talent and luck are independent, and both raise the chance of being hired. Among the hired, learning that someone had little talent implies they had luck, because something got them in. A negative correlation appears between two independent variables, generated entirely by the selection.

graph TD
    subgraph Fork["Fork: adjust"]
        F1["Stone size"] --> F2["Treatment"]
        F1 --> F3["Recovery"]
    end
    subgraph Chain["Chain: do not adjust for total effect"]
        C1["Drug"] --> C2["Blood pressure"]
        C2 --> C3["Outcome"]
    end
    subgraph Collider["Collider: never adjust"]
        D1["Talent"] --> D2["Hired"]
        D3["Luck"] --> D2
    end

    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class F1,F2,F3 emerald
    class C1,C2,C3 amber
    class D1,D2,D3 rose

This is not a curiosity. Conditioning on a collider is what happens every time an analysis is restricted to a selected population, and selection is the normal condition of observational data.

Griffith and colleagues demonstrated the consequences at scale during the COVID-19 pandemic. Studies of risk factors drawn from people who had been tested, hospitalised, or who volunteered for a study are conditioning on a variable that many risk factors affect. The paper shows that this can induce associations between exposures and outcomes that do not exist in the general population, and in some configurations reverse their sign (Griffith et al., 2020, Collider bias undermines our understanding of COVID-19 disease risk and severity, Nature Communications 11, 5749).

The same structure appears in every product analytics question that starts "among users who converted" or "for accounts that completed onboarding".

[IMAGE: Diagram of the adjustment formula with its two halves annotated separately. The inner conditional expectation is bracketed and labelled "what a regression computes"; the outer sum over the marginal distribution of Z is bracketed and labelled "the reweighting that makes it causal". A small inset shows the kidney stone caseload mix to illustrate what the outer weighting corrects. Caption: "The regression is half the formula. The half usually missing is the reweighting."]

The Backdoor Criterion, and the Four Bad Controls

Given a graph, the question of which variables to adjust for has a mechanical answer. A set \(Z\) identifies the causal effect of \(X\) on \(Y\) if:

  1. \(Z\) blocks every path from \(X\) to \(Y\) that begins with an arrow into \(X\) (the backdoor paths), and
  2. \(Z\) contains no descendant of \(X\).

When it holds, the effect is given by the adjustment formula:

\[P(Y \mid do(X = x)) = \sum_{z}P(Y \mid X = x, Z = z)\,P(Z = z)\]

Read the two halves separately, because the difference between them is the entire difference between association and causation. Inside the sum, you condition on \(Z\), which is what a regression does. Outside, you weight by the marginal distribution of \(Z\), not by its distribution among the treated. That reweighting asks what would happen if the whole population, with its actual covariate mix, received the treatment. The kidney stone reversal is exactly the failure to do this reweighting: the pooled numbers weight each treatment's outcome by its own caseload mix instead of by the population's.

Four adjustments are actively harmful, and the "kitchen sink" regression produces all four.

A mediator, when the total effect is the target. Removes the pathway you were measuring.

A descendant of the outcome. Conditions on a collider, opening a spurious path.

A pure cause of treatment (an instrument). This is the least intuitive and the most common. Adjusting for a variable that affects treatment and not the outcome removes variation in \(X\) that was clean, leaving a higher proportion of the remaining variation attributable to unmeasured confounders. The bias from that confounding is therefore amplified, and the variance rises at the same time. Adding a strong predictor of treatment to a regression can make the estimate worse in both senses simultaneously.

Any post-treatment variable in a randomised experiment. Randomisation guarantees independence between assignment and potential outcomes at the moment of assignment. Restricting to users who completed onboarding, an event the treatment affects, discards that guarantee and reintroduces selection.

[IMAGE: Four small DAGs in a 2x2 grid, each with the "bad control" node circled in red and an arrow annotation explaining the damage. (a) mediator: "removes the effect being measured". (b) descendant of outcome: "opens a collider path". © instrument: "amplifies residual confounding bias and inflates variance". (d) post-treatment in an RCT: "discards randomisation". Caption: "Four adjustments that make the estimate worse, all produced by throwing every column into the regression."]

By the Numbers

Quantity Value Source
Kidney stones, small: open surgery vs percutaneous 93% (81/87) vs 87% (234/270) Charig et al. (1986)
Kidney stones, large: open surgery vs percutaneous 73% (192/263) vs 69% (55/80) Charig et al. (1986)
Kidney stones, pooled 78% (273/350) vs 83% (289/350) Charig et al. (1986)
Caseload imbalance driving the reversal 263 of 350 open-surgery cases were large stones; 270 of 350 percutaneous cases were small Charig et al. (1986)
Year the reversal was first described 1903 (Yule); formalised 1951 (Simpson) historical
Gap between the backdoor criterion and a published "bad controls" taxonomy roughly 30 years Pearl (1995) to Cinelli et al. (2024)
Effect of collider bias in selected COVID samples can induce, and in some configurations reverse, exposure-outcome associations Griffith et al. (2020)
Cinelli et al. page range Sociological Methods & Research 53(3), 1071-1104 Cinelli et al. (2024)

Sources: Charig et al., 1986, BMJ 292:879-882; Griffith et al., 2020, Nature Communications 11:5749; Cinelli, Forney and Pearl, 2024. The kidney stone figures are as reproduced in the standard presentation of the study and are consistent across sources; the "roughly 30 years" row is a characterisation of the literature, not a measured quantity.

A Concrete Example

A subscription product wants to know whether its onboarding checklist causes retention. Users who complete the checklist retain at 71%; users who do not retain at 44%. The 27-point gap is the number in the deck.

The analyst has four covariates available and runs the obvious regression with all of them.

Variable What it is Causal role
acquisition_channel organic, paid, referral affects both checklist completion and retention
signup_month cohort affects both
activated_day_1 performed a core action on day one caused by the checklist, and affects retention
support_ticket_filed opened a ticket in month one caused by both engagement and by dissatisfaction

Step 1: the correct adjustments. acquisition_channel and signup_month are forks. Paid users complete fewer checklists and retain worse; both effects run through the same variable. Adjusting for them closes real backdoor paths. Suppose doing so alone moves the estimate from 27 points to 19 points. That 8-point reduction is confounding removed, and it is what "controlling for confounders" is supposed to do.

Step 2: the mediator. activated_day_1 is downstream of the checklist. The checklist works partly by getting users to perform that action. Adding it to the regression drops the estimate from 19 points to 6 points. The naive reading is "most of the effect was explained away". The correct reading is that 13 points of the effect flow through activation, and by conditioning on activation the model now reports only the direct effect. If the decision is whether to keep the checklist, the total effect of 19 is the relevant quantity and the 6 is a different question that nobody asked.

Step 3: the collider. support_ticket_filed is caused by engagement, which the checklist increases, and by dissatisfaction, which reduces retention. It is a common effect of a cause and something related to the outcome. Adding it opens a path: among ticket-filers, high engagement implies lower dissatisfaction, so the checklist looks better than it is. The estimate moves from 6 points back up to 9.5. The direction is not predictable without the graph, and the number moved by 3.5 points because of a variable that no reasonable person would defend as a confounder if asked directly.

Step 4: what the four models say.

Adjustment set Estimate What it means
None 27.0 pts confounded by channel and cohort
Channel + month 19.0 pts the total effect, correctly identified
Channel + month + activation 6.0 pts direct effect only; mediator removed 13 pts of real effect
All four 9.5 pts direct effect plus 3.5 pts of collider-induced bias

Only the second row answers "should we keep the checklist". The fourth row, the one produced by the default habit of including every available column, is wrong in two ways at once and its confidence interval is no wider than the others'. Nothing in the regression output flags it.

Step 5: what would have caught it. One question per covariate: is this measured before the treatment, or after? activated_day_1 and support_ticket_filed are both after. That single rule, applied mechanically, removes both bad controls without requiring anyone to draw a graph, and it is why "adjust only for pre-treatment variables" is the most valuable heuristic available to someone who will not do the full analysis.

[IMAGE: Horizontal dot-and-interval plot with four rows, one per adjustment set, showing point estimates at 27.0, 19.0, 6.0 and 9.5 percentage points with identically sized confidence intervals. A vertical dashed line marks the true total effect at 19.0. Annotations point to the mediator-induced drop and the collider-induced rebound. Caption: "Four defensible-looking regressions, one correct answer, and no way to tell from the output."]

Where It Breaks

The graph is an assumption, and it is the strongest one you make

Everything above is conditional on a stated structure. Two analysts with different graphs get different adjustment sets and different answers from identical data, and no statistical procedure adjudicates between them. Causal discovery algorithms recover only an equivalence class of graphs consistent with the observed conditional independences, and that class is usually large.

This is sometimes raised as an objection to the framework, which gets it backwards. The assumptions were always there; the graph makes them visible and their consequences computable. A regression with a hand-picked control set encodes exactly the same assumptions and hides them.

Unmeasured confounding is invisible on the graph you drew

The backdoor criterion certifies a set relative to the asserted structure. A confounder you did not draw is not detected as missing; it is simply absent from the analysis, and the resulting estimate is biased with no warning of any kind.

The honest response is sensitivity analysis: quantify how strongly an unmeasured confounder would have to be associated with both treatment and outcome to overturn the conclusion. If the answer is "more strongly than any measured covariate", the finding is robust; if it is "about as strongly as the weakest covariate you did include", it is not. This converts an untestable assumption into a stated fragility and belongs in every observational report.

Selection into the dataset is a collider you cannot condition away

If the population you observe was already filtered by something both the treatment and the outcome affect, the bias is baked in before any analysis begins, and no choice of adjustment set repairs it. A dataset of customers who did not churn, a log of sessions that completed, a cohort of patients who were tested: each is a conditioned collider, and the correction requires either modelling the selection mechanism explicitly or obtaining unselected data.

Identification is not estimation

Satisfying the backdoor criterion tells you the effect is a function of observable distributions. Computing that function still requires positivity, that some units at every covariate value plausibly receive each treatment, a correct functional form or a flexible enough estimator, and enough data. Positivity failures in particular are silent: if a subgroup is always treated, the estimator extrapolates from a model rather than adjusting, and most software reports nothing.

With many covariates the adjustment formula has no cells to sum over and becomes a regression or a weighting estimator, at which point specification error re-enters through the door the graph was supposed to close. Double machine learning exists to bound that damage, using Neyman-orthogonal scores so first-order errors in the nuisance models do not propagate, plus cross-fitting so regularisation bias does not correlate with the score (Chernozhukov et al., 2018, Econometrics Journal 21(1), C1-C68).

Some questions are not do-questions

"Would this user have converted had they not seen the banner", asked about a specific user who did see it, is a counterfactual about a world that contradicts the observed one. It sits on a higher rung than intervention and generally requires assumptions about the functional form of the structural equations, not just the graph. Interventional identification does not deliver it, and treating a personalised uplift score as if it did overstates what the model knows.

Alternative Designs

Approach What it assumes Key advantage Key limitation Best when
Randomisation Nothing beyond correct execution Balances unmeasured covariates in expectation Often impossible, unethical, or slow You control assignment
Backdoor adjustment All confounders measured; positivity Uses existing data; mechanical given a graph Silent failure under unmeasured confounding Strong domain knowledge of the mechanism
Propensity weighting Same as backdoor Collapses many covariates to one score Extreme weights destabilise; same ignorability assumption High-dimensional covariates, decent overlap
Instrumental variables Exclusion and independence, both untestable Survives unmeasured confounding Identifies only the complier effect; weak instruments are worse than none A credible as-good-as-random source of variation exists
Difference-in-differences Parallel counterfactual trends Handles time-invariant unobserved confounders Untestable assumption; staggered adoption breaks naive estimators Panel data with a clean intervention date
Regression discontinuity Continuity at the cutoff; no manipulation Local randomisation is highly credible Estimate applies only at the threshold Treatment assigned by a threshold rule
Front-door Fully mediated effect; clean mediator Works with an unmeasured confounder on \(X \to Y\) Conditions rarely hold cleanly The full mechanism is observed
Double machine learning Backdoor plus rate conditions on nuisances Flexible ML with valid confidence intervals Cross-fitting is mandatory; still needs ignorability Many covariates, backdoor plausible

The row that deserves more use than it gets is randomised encouragement, a variant of the instrumental variables row. When the treatment itself cannot be randomised, randomising an invitation, prompt or default produces an instrument that is valid by construction, converting two untestable assumptions into design properties. It is available far more often than teams assume.

How It Is Used in Practice

Feature selection is an adjustment-set decision in disguise. A predictive model that includes post-treatment features will predict well and its coefficients will not answer any causal question. Teams that use the same model for prediction and for "which levers matter" are reading a causal claim off an object that was never built to support one. The two use cases need two models, or at least two feature sets.

Product analytics defaults to collider conditioning. "Among users who converted", "for sessions longer than 30 seconds", "for accounts that reached step three" are all conditioning on selection. Sometimes it is exactly right, when the question genuinely concerns that subpopulation and the treatment does not affect membership. When treatment affects who is in the group, it is a collider and the comparison is manufactured.

Observational studies in ML evaluation have the same structure. Comparing models by their performance on the subset of queries where both returned results, or on the users who engaged, conditions on an outcome-adjacent variable. The resulting comparison is not the comparison intended.

Segment reporting is Simpson's paradox waiting to happen. Any metric whose population composition shifts can decline in every segment while rising overall, or the reverse. This is why segment-level reporting is not optional for aggregate KPIs, and why a top-line trend can be both true and completely misleading.

stateDiagram-v2
    [*] --> Question
    Question: State the causal question precisely
    Question --> Graph
    Graph: Draw the structure, unmeasured nodes included
    Graph --> Backdoor
    Backdoor: Apply the backdoor criterion
    Backdoor --> Identified: a valid set exists
    Backdoor --> Alternative: no valid set
    Alternative: IV, front door, RD, DiD, or decline
    Alternative --> Identified
    Identified --> Positivity
    Positivity: Check overlap on the adjustment set
    Positivity --> Estimate: adequate overlap
    Positivity --> Restrict: poor overlap
    Restrict: Restrict to the overlap region and report it
    Restrict --> Estimate
    Estimate: Regression, weighting, or DML with cross-fitting
    Estimate --> Sensitivity
    Sensitivity: How strong must an unmeasured confounder be?
    Sensitivity --> [*]

The discipline that pays for itself is running the first three states before opening a notebook. Most bad-control errors are settled by a conversation about what causes what, and settling them afterwards means discovering that a shipped analysis was wrong.

[IMAGE: Split-screen comparison of two analytics dashboards showing the same metric over twelve months. Left panel: a single aggregate line trending upward. Right panel: four segment lines all trending downward, with a stacked area chart beneath showing the population mix shifting toward the highest-value segment. Caption: "Both charts are correct. Only one of them supports a conclusion."]

[IMAGE: Flowchart of the single temporal heuristic applied to six example covariates. Each passes through a gate reading "measured strictly before treatment?", with pre-treatment variables routed to an "eligible for adjustment" bin and post-treatment ones routed to an "exclude" bin, each annotated with the bad-control category it would otherwise have been. Caption: "One question per covariate removes three of the four bad controls without drawing a graph."]

Insights Worth Remembering

  1. Adjustment is not a safety measure. More controls is not more careful. Adding a collider introduces bias that was absent, adding a mediator removes the effect being measured, and adding an instrument amplifies whatever confounding remains. Safety comes from the right set, not the large one.

  2. The data cannot arbitrate between the pooled and the stratified answer. Both are arithmetically valid. The choice is settled by the causal role of the grouping variable, which is knowledge you bring rather than knowledge you extract.

  3. Colliders invert the intuition, which is why they are missed. Conditioning normally removes association; on a collider it creates one. Every restriction to a selected population, in a product funnel or a clinical sample, is this operation performed without noticing.

  4. The single most valuable heuristic is temporal, not graphical. Adjust only for variables realised before the treatment. It requires no graph, catches mediators, outcome descendants and post-treatment conditioning in experiments, and is the rule that survives contact with an analyst in a hurry.

  5. Instrument-like controls amplify bias and inflate variance simultaneously. Adding a strong predictor of treatment that has no path to the outcome makes the estimate worse in both senses. This is the least known of the four bad controls and the most likely to be added deliberately, because it "explains treatment well".

  6. Identification precedes estimation, and sample size cannot substitute. If the effect is not identified, a million more rows produce a tighter interval around the wrong quantity. Settling identification first is the discipline the framework actually imposes.

  7. Drawing the graph does not add assumptions; it exposes the ones you already made. A regression with a hand-picked control set encodes an identical set of causal claims and hides them in the covariate list. The objection that graphs require assumptions is an objection to the visibility, not to the requirement.

  8. Sensitivity analysis is the honest ending for observational work. Since unmeasured confounding cannot be detected, the reportable quantity is how strong it would have to be to change the conclusion. A result robust only to a confounder weaker than the ones you measured is not a result.

Open Questions

How much of applied regression practice is affected? It is established that bad controls bias estimates and that the kitchen-sink habit is widespread. What is not established is the aggregate magnitude across published applied work, because re-analysing a study requires the causal structure that the original rarely states. Attempts to audit this systematically run into the same problem the framework identifies: the graph is not in the paper.

Can bad controls be detected automatically? Given a graph, the check is mechanical; without one, a covariate's role is not identifiable from its distribution. Whether partial structural knowledge, temporal ordering plus a few asserted edges, suffices to flag most bad controls in practice is an open engineering question with real practical value.

How should sensitivity analysis be standardised? Several frameworks exist (E-values, Rosenbaum bounds, omitted-variable-bias benchmarking), they answer subtly different questions, and there is no convention about which to report. The absence of a standard is a large part of why sensitivity analysis remains optional in practice.

Do LLM-assisted causal graphs help or launder? Language models can propose plausible DAGs from a domain description, and it is measured that they produce structures experts find reasonable. Whether that improves analyses or simply supplies confident-looking justification for whatever adjustment set was going to be used is not established, and the failure mode, an authoritative-looking graph nobody interrogated, would be difficult to detect.

What is the right treatment of selection in ML evaluation? Model comparisons routinely condition on outcome-adjacent filters, and the causal literature's tools have not been systematically applied to benchmark design. Whether standard leaderboard comparisons carry collider bias of practical size is, as far as this author can find, largely unexamined.

Sources and Further Reading

  1. Cinelli, C., Forney, A., & Pearl, J. (2024). "A Crash Course in Good and Bad Controls." Sociological Methods & Research, 53(3), 1071-1104. DOI
  2. Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press. DOI
  3. Griffith, G. J., Morris, T. T., Tudball, M. J., et al. (2020). "Collider bias undermines our understanding of COVID-19 disease risk and severity." Nature Communications, 11, 5749. DOI
  4. Charig, C. R., Webb, D. R., Payne, S. R., & Wickham, J. E. (1986). "Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy." British Medical Journal, 292(6524), 879-882. DOI
  5. Rubin, D. B. (1974). "Estimating causal effects of treatments in randomized and nonrandomized studies." Journal of Educational Psychology, 66(5), 688-701. DOI
  6. Holland, P. W. (1986). "Statistics and Causal Inference." Journal of the American Statistical Association, 81(396), 945-960. DOI
  7. Rosenbaum, P. R., & Rubin, D. B. (1983). "The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika, 70(1), 41-55. DOI
  8. Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). "Double/debiased machine learning for treatment and structural parameters." The Econometrics Journal, 21(1), C1-C68. DOI
  9. Imbens, G. W., & Angrist, J. D. (1994). "Identification and Estimation of Local Average Treatment Effects." Econometrica, 62(2), 467-475. DOI
  10. Hernán, M. A., & Robins, J. M. (2020). Causal Inference: What If. Chapman & Hall/CRC. Free PDF
  11. Wager, S., & Athey, S. (2018). "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests." JASA, 113(523), 1228-1242. DOI
  12. Gelman, A., & Loken, E. (2013). "The Garden of Forking Paths." Columbia University working paper. PDF

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.