Reasoning & Evaluation

The Ladder of Causation: Why No Amount of Observational Data Climbs It Alone

In 2023 GPT-4 scored 97% on a classic cause-and-effect benchmark and 62% on one that hands it the causal graph and asks it to compute. Both results fit a theorem proved in 2020: data from one rung of Pearl's ladder almost never determines the answers on the rung above. Here is what the hierarchy says, how structural causal models climb it by paying in assumptions, and what that implies for claims that language models reason causally.

Praveen T N 24 min read Contains maths Diagrams

In April 2023 a team from Microsoft Research and the University of Chicago reported that GPT-4 picked the direction of causation correctly on 97% (weighted) of the Tübingen cause-effect pairs, 13 points above the best statistical algorithm on that benchmark (Kıcıman et al., 2023, arXiv:2305.00050). Months later a NeurIPS paper gave GPT-4 ten thousand causal questions with the graph and every needed probability in the prompt, so everything had to be computed. It scored 62.03%, against a random baseline of 49.27% (Jin et al., 2023, CLadder, arXiv:2312.04350).

The numbers are not in tension. The first asks which way causation runs between altitude and temperature, a fact people have written down; the second asks the model to execute causal inference. Pearl's Ladder of Causation separates those questions, and the Causal Hierarchy Theorem turns the ladder from a metaphor into mathematics: information at one rung almost never determines answers at the rung above.

Why this matters: Every A/B readout, uplift model, liability argument and "the model reasons causally" claim sits on a rung that fixes which inputs are logically required. Confuse the rungs and you ship "what would have happened" scores no experiment could validate, or mistake memorised causal facts for causal inference. The hierarchy tells you, before you collect a row, whether more data can help at all.

TL;DR

The Causal Hierarchy Theorem (2020) proves that structural causal models in which association \(P(y \mid x)\), intervention \(P(y \mid do(x))\) or counterfactual \(P(y_x \mid x', y')\) collapses to a lower layer form a Lebesgue measure-zero set. Layer 2 never collapses to Layer 1 over unrestricted models.
Climbing costs assumptions, not data. A graph buys Layer 2 from Layer 1, and the do-calculus is complete (2006, proved twice independently): if its three rules cannot derive an effect, nothing can without new assumptions.
Counterfactuals also need functional form. In the worked example, one student and one unchanged population yield counterfactual scores of 159, 165 or 177 depending on an untestable noise assumption.
Two binary models can match on every observational and experimental probability while their probability of necessity and sufficiency is 0 and 0.5. The Tian-Pearl bounds are exactly [0, 0.5], both ends attained.
LLM evidence splits along the ladder: 97% where causal facts live in text (with GPT-4 reproducing 61% of the Tübingen cells from memory), near chance on Corr2Cause's 200K computed questions, and mean drops of 6.7 to 29.6 points across three benchmarks when variable names are anonymised (2026 preprint).

At a Glance

flowchart LR
    D1["Rung 1 data: passive observations"] --> Q1["Association: seeing"]
    A2["Causal graph assumptions"] --> E2["Identification: do-calculus"]
    Q1 --> E2
    E2 --> Q2["Intervention: doing"]
    A3["Functional form and noise assumptions"] --> E3["Abduction, action, prediction"]
    Q2 --> E3
    E3 --> Q3["Counterfactual: imagining"]
    CHT["Causal Hierarchy Theorem: no free climb"] -.-> E2
    CHT -.-> E3

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class D1 blue
    class A2,A3 amber
    class E2,E3 purple
    class Q1,Q2,Q3 teal
    class CHT rose

Each step up needs an amber input that is not data; the rose theorem makes those inputs mandatory.

From Path Coefficients to a Hierarchy Theorem

Sewall Wright read causal effects off arrows between variables long before anyone spoke of a ladder (Wright, 1934, Annals of Mathematical Statistics 5(3)). Neyman wrote potential yields for agricultural plots in 1923, and Rubin extended the notation to nonrandomised studies (Rubin, 1974, Journal of Educational Psychology 66(5)).

The lineages met formally in the 1990s. Balke and Pearl evaluated counterfactuals by treating the antecedent as an intervention on a functional model, and bounded them when only probabilities were known (Balke and Pearl, 1994, arXiv:1302.6784). Pearl's Biometrika paper introduced the do-calculus, three graphical rules that rewrite interventional expressions into observational ones (Pearl, 1995, Biometrika 82(4)). Tian and Pearl gave sharp bounds on probabilities of causation from experimental and observational data (Tian and Pearl, 2000, Annals of Mathematics and AI 28).

Whether the do-calculus was enough stayed open until 2006, when two groups proved completeness independently (Shpitser and Pearl, 2006, AAAI; Huang and Valtorta, 2006, arXiv:1206.6831). Shpitser and Pearl then extended the characterisation to counterfactuals (Shpitser and Pearl, 2008, JMLR 9).

timeline
    title From arrows to a theorem about the ladder
    1934 : Wright publishes The Method of Path Coefficients
    1974 : Rubin extends Neyman potential outcomes to nonrandomised studies
    1994 : Balke and Pearl compute and bound counterfactual probabilities
    1995 : Pearl introduces the do-calculus in Biometrika
    2006 : Shpitser and Pearl, and Huang and Valtorta, prove do-calculus complete
    2018 : The Book of Why names the three-rung Ladder of Causation
    2020 : Bareinboim, Correa, Ibeling and Icard prove the Causal Hierarchy Theorem
         : Imbens publishes the potential outcomes versus DAG critique
    2023 : Kiciman et al., Zecevic et al. and CLadder test LLMs on the ladder

The Book of Why (Pearl and Mackenzie, 2018) named the ladder, and Pearl's Communications of the ACM essay drew the machine-learning moral: "as long as our system optimizes some property of the observed data, however noble or sophisticated, while making no reference to the world outside the data, we are back to level-1" (Pearl, 2019, CACM 62(3), author version R-481).

The proof arrived as Columbia report R-60 in 2020, later an ACM Books chapter (Bareinboim et al., 2022). Ibeling and Icard gave a topological version: assumption-free causal inference works only on a meagre set of models (Ibeling and Icard, 2021, arXiv:2107.08558), and Xia, Lee, Bengio and Bareinboim showed the theorem binds neural models: expressiveness does not buy Layer 2 from Layer 1 data (Xia et al., 2021, arXiv:2107.00793).

[IMAGE: A vertical three-rung ladder. Each rung carries its symbol, its verb (seeing, doing, imagining) and one question: "What does a symptom tell me about a disease?", "What if I take aspirin?", "Was it the aspirin that stopped my headache?". Amber brackets between rungs read "graph assumptions" and "functional-form assumptions". Caption: "Each step up Pearl's hierarchy is paid for with assumptions the lower rung's data cannot supply."]

How Structural Causal Models Climb the Ladder

The object

A structural causal model \(M = \langle U, V, F, P(U) \rangle\) has exogenous background variables \(U\) with distribution \(P(U)\), observed endogenous variables \(V\), and one function per endogenous variable:

\[ V_i := f_i(\mathrm{PA}_i, U_i). \]

The := is assignment. \(Y := f(X, U_Y)\) and \(X := g(Y, U_X)\) can generate the same joint distribution, but only one describes the mechanism, an asymmetry regression lacks. Fixing \(u\) fixes every variable, so all uncertainty enters through \(P(U)\); that is what lets one object generate all three rungs.

Three layers

Layer 1 pushes \(P(U)\) through \(F\): \(P^M(v) = \sum_{u : V(u) = v} P(u)\).

Layer 2 uses the submodel \(M_x\), in which the equation for \(X\) is replaced by the constant \(x\):

\[ P^M(y \mid do(x)) = \sum_{u \,:\, Y_{M_x}(u) = y} P(u). \]

With independent noise terms this reduces to the truncated factorisation, which drops the factor of each intervened variable:

\[ P(v \mid do(x)) = \prod_{i \,:\, V_i \notin X} P(v_i \mid \mathrm{pa}_i) \Big|_{X = x}. \]

Layer 3 holds joint distributions over potential responses \(Y_x(u)\), the value of \(Y\) in \(M_x\) under background \(u\). The canonical query conditions on the actual world and asks about another:

\[ P(Y_x = y \mid X = x', Y = y') = \sum_{u} \mathbf{1}\big[Y_x(u) = y\big] \, P(u \mid x', y'). \]

The subscript \(x\) and evidence \(x'\) may contradict each other: "the patient took the drug and died; would they have lived without it?" places one unit in two worlds, which no experiment can.

[IMAGE: Twin-network diagram. The factual network X, M, Y on the left and the counterfactual copy with X fixed on the right share exogenous nodes U_M and U_Y drawn between them. Caption: "A counterfactual joins two worlds through shared background variables, which is exactly what no experiment observes."]

The Causal Hierarchy Theorem

Call two SCMs \(\mathcal{L}_i\)-equivalent if they agree on every Layer-\(i\) quantity. Layer \(j\) collapses to layer \(i < j\) relative to \(M^*\) if every model \(\mathcal{L}_i\)-equivalent to \(M^*\) is also \(\mathcal{L}_j\)-equivalent. The theorem states

\[ \mu\big(\{ M \in \Omega : \text{some layer collapses relative to } M \}\big) = 0, \]

with \(\mu\) the Lebesgue measure over a suitable encoding of \(\mathcal{L}_3\)-equivalence classes (Bareinboim et al., 2022, Theorem 1). Draw an SCM at random under any smooth distribution and the probability that the hierarchy collapses is zero. The authors' corollary: "to answer questions at Layer i, one needs knowledge at Layer i or higher."

The halves differ in strength. Layer 2 never collapses to Layer 1: for any model with a non-trivial dependence, a second model with one exogenous variable driving everything reproduces the joint distribution while interventions do nothing. For Layers 2 and 3, the proof swaps potential responses of two background settings with small probability, preserving every \(P(y \mid do(x))\) while changing cross-world probabilities. The theorem formalises Cartwright's "no causes in, no causes out": it rules out an assumption-free climb, not a climb whose missing information arrives from outside the data.

Buying rung two: do-calculus

A causal graph \(G\) is Layer-2 information: which variables are absent from which functions, and which noise terms are shared. Write \(G_{\overline{X}}\) for \(G\) with arrows into \(X\) removed and \(G_{\underline{Z}}\) with arrows out of \(Z\) removed. The rules are:

\[ \text{R1: } P(y \mid do(x), z, w) = P(y \mid do(x), w) \quad \text{if } (Y \perp\!\!\!\perp Z \mid X, W)_{G_{\overline{X}}} \]

\[ \text{R2: } P(y \mid do(x), do(z), w) = P(y \mid do(x), z, w) \quad \text{if } (Y \perp\!\!\!\perp Z \mid X, W)_{G_{\overline{X}\underline{Z}}} \]

\[ \text{R3: } P(y \mid do(x), do(z), w) = P(y \mid do(x), w) \quad \text{if } (Y \perp\!\!\!\perp Z \mid X, W)_{G_{\overline{X}\,\overline{Z(W)}}} \]

where \(Z(W)\) is the set of \(Z\) nodes that are not ancestors of any \(W\) node in \(G_{\overline{X}}\). Rule 1 inserts or deletes observations, Rule 2 swaps an action for an observation, Rule 3 inserts or deletes actions. An effect is identified when the rules remove every \(do\). Backdoor and front-door adjustment are special cases, treated in the companion post on bad controls, colliders and mediators; this post takes identification as given and looks above it.

Completeness turns the rules into a boundary. The ID algorithm returns an estimand or a hedge, a witness of two models that agree observationally and disagree on the effect. More observational rows cannot help, because the hedge models agree on every one.

Buying rung three: abduction, action, prediction

Counterfactuals need the functions and a posterior over background variables. Pearl's recipe for \(P(Y_x \mid e)\):

Abduction. Update \(P(u)\) to \(P(u \mid e)\).
Action. Replace the equations for \(X\) with \(X := x\).
Prediction. Push \(P(u \mid e)\) through the modified model.

With invertible noise the answer is a number; without known functions, often only a bound. The canonical bounded quantities are the probabilities of causation: necessity \(PN = P(Y_{x'} = y' \mid X = x, Y = y)\), the legal "but-for" question, and necessity-and-sufficiency \(PNS = P(y_x, y'_{x'})\). From experiments alone, Tian and Pearl show

\[ \max\{0,\; P(y_x) - P(y_{x'})\} \;\le\; PNS \;\le\; \min\{P(y_x),\; P(y'_{x'})\}, \]

narrowed by adding observational data and point-identified under monotonicity (treatment harms no one). The interval's width is the CHT in numbers.

Seeing It in Motion

Where the inference engine refuses

flowchart TB
    Q["Query: effect or counterfactual"] --> ENG["Identification engine"]
    G["Causal graph"] --> ENG
    FF["Functional form, monotonicity"] --> ENG
    DATA["Observational or experimental data"] --> EST["Estimator"]
    ENG -->|"identified"| EST
    ENG -->|"hedge found"| NID["Not identifiable from these inputs"]
    NID --> MORE["Add assumption or run experiment"]
    NID --> BND["Report partial identification bounds"]
    MORE --> ENG
    EST --> ANS["Estimate with uncertainty"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class Q,DATA blue
    class G,FF amber
    class ENG,EST purple
    class ANS,BND teal
    class NID rose
    class MORE slate

Identification is decided on assumptions alone, before data enters.

One counterfactual, step by step

sequenceDiagram
    participant E as Evidence for one unit
    participant AB as Abduction
    participant M as Structural model
    participant AC as Action
    participant PR as Prediction
    E->>AB: X=4, M=22, Y=120
    AB->>M: Invert equations for this unit
    M-->>AB: U_M=4 and U_Y=8
    AB->>AC: Hold U at posterior values
    AC->>M: Replace equation for X with X=10
    M->>PR: Modified model plus unit noise
    PR->>PR: Recompute M then Y in order
    PR-->>E: Counterfactual Y equals 165
    Note over AB,PR: Valid only if the same U applies in both worlds

The worked example puts numbers on that note.

[IMAGE: Three-panel strip. Panel 1: observed values flow backwards to solve for U_M and U_Y. Panel 2: the arrow into X is cut, X pinned to 10. Panel 3: the fixed U values flow forward to a new Y. Caption: "Abduction runs the model backwards for one unit, action edits it, prediction runs it forwards with that unit's noise held fixed."]

By the Numbers

Result	Rung	Information supplied?	Number	Comparison
CHT, models where any layer collapses	all	n/a	measure zero	Layer 2 to 1 collapse never occurs
Do-calculus completeness	2	graph	3 rules suffice	proved twice, 2006
Tübingen pairs, GPT-4, weighted	2 as recall	variable names	97%	best prior algorithm 83%
Tübingen memorisation probe, GPT-4	none	dataset name	61% of cells, 25% of rows	GPT-3.5: 58% and 19%
CRASS counterfactuals, GPT-4	3, commonsense	scenario text	92%	20 points above prior best
CLadder overall, GPT-4	1 to 3 computed	graph and numbers	62.03%	random 49.27%, GPT-3.5 52.18%
CLadder by rung, GPT-4	1 / 2 / 3	graph and numbers	63.01 / 62.82 / 60.55%	nearly flat
CLadder by rung, GPT-4 plus CausalCoT	1 / 2 / 3	graph and numbers	83.35 / 67.47 / 62.05%	overall 70.40%
Corr2Cause, 17 LLMs	1 to graph, computed	correlational statements	close to random	over 200K samples
Caliper, five models 3.8B to 14B	2 computed; 3 commonsense	anonymised names	mean drops of 7.6 / 27.0 / 11.1 points on CLadder / CRASS / e-CARE	nine models of 70B and above: 6.7 / 29.6 / 18.1

Sources: theory rows from references 1, 3 and 4; Tübingen, memorisation and CRASS from Kıcıman et al., 2023; CLadder from Jin et al., 2023; Corr2Cause from Jin et al., 2024, ICLR; Caliper from Yu and Zhou, 2026, a preprint not peer reviewed as of September 2026. LLM figures describe specific 2023 to 2026 model versions, not current frontier models.

Where no causal information is supplied, GPT-4 excels, which is what recall looks like (older base models such as ada scored at chance). With the graph and numbers supplied, GPT-4 is nearly flat across rungs although Layer 3 is strictly harder; an inference engine would degrade with rung, a surface pattern-matcher would not. CausalCoT, which makes the model extract graph and query first, lifts rung 1 by 20 points and rung 3 by 1.5.

[IMAGE: Grouped bar chart of CLadder accuracy by rung for random, GPT-3.5, GPT-4 and GPT-4 plus CausalCoT, with a dashed line at 50%. Annotate GPT-4's flat profile. Caption: "With the graph given, the difficulty is computation, and scaffolding helps least where the ladder is highest."]

A Concrete Example

Part 1: same experiments, opposite verdicts

Binary \(X \sim \text{Bernoulli}(0.5)\) and \(U \sim \text{Bernoulli}(0.5)\), independent. Model A: \(Y := U\). Model B: \(Y := X \oplus U\).

Layer 1. \(P(Y{=}1 \mid X{=}x) = 0.5\) in both. Identical joints.
Layer 2. Intervening on \(X\) leaves \(U\) alone, so \(P(Y{=}1 \mid do(x)) = 0.5\) in both. A million-unit RCT cannot tell them apart.
Layer 3. Observe \(X{=}1, Y{=}1\); would \(Y\) be 1 had \(X\) been 0? A: abduction gives \(U{=}1\), so \(Y_0 = 1\) and \(PN = 0\). B: abduction gives \(U{=}0\), so \(Y_0 = 0\) and \(PN = 1\).
Bounds. \(\max\{0, 0.5 - 0.5\} = 0 \le PNS \le \min\{0.5, 0.5\} = 0.5\). A has \(PNS = 0\); B has \(PNS = P(U{=}0) = 0.5\). Both ends attained: the experiment leaves the whole interval open.

Part 2: a linear SCM, one student, three answers

\(X\) is weekly tutoring hours, \(M\) a practice-test score, \(Y\) the exam score, noise zero-mean:

\[ M := 10 + 2X + U_M, \qquad Y := 40 + 1.5X + 3M + U_Y. \]

A student logged \(X = 4\), scored \(M = 22\) and \(Y = 120\). What would they have scored with \(X = 10\)?

Step 1, abduction.

\[ U_M = 22 - 10 - 8 = 4, \qquad U_Y = 120 - 40 - 6 - 66 = 8. \]

Step 2, action. Set \(X := 10\); keep \(U_M = 4\), \(U_Y = 8\).

Step 3, prediction.

\[ M_{10} = 10 + 20 + 4 = 34, \qquad Y_{10} = 40 + 15 + 102 + 8 = 165. \]

Step 4, compare with Layer 2. \(E[Y \mid do(X{=}4)] = 40 + 6 + 54 = 100\) and \(E[Y \mid do(X{=}10)] = 40 + 15 + 90 = 145\), an average effect of 45. The student's gain is also \(165 - 120 = 45\); they sit 20 points above the mean in both worlds because \(U_Y + 3U_M = 20\) carries across.

That equality is not a finding. Additive noise with a common slope forces every unit's effect to equal the average. Now let the slope vary by student, \(Y := 40 + (1.5 + U_S)X + 3M + U_Y\) with \(E[U_S] = 0\) and \(U_S\) independent of the rest. Every interventional mean is unchanged, so no experiment separates this model from the first.

Step 5, abduction again. One equation, two unknowns: \(4U_S + U_Y = 8\).

Assumed \(U_S\)	Implied \(U_Y\)	\(Y_{10}\)	Individual effect
0	8	40 + 15 + 102 + 8 = 165	45
2	0	40 + 35 + 102 + 0 = 177	57
minus 1	12	40 + 5 + 102 + 12 = 159	39

Same student, population and experiments: 159, 165 or 177. The first model silently assumed one noise term held constant across worlds, and that selected 165. This is nearly Guido Imbens's objection to the salary counterfactual in The Book of Why: the recipe "implicitly assumes" the residual under one treatment equals the residual under the other, which amounts to preserving a unit's rank across treatments, an assumption economists make explicitly and argue for (Imbens, 2020, JEL 58(4), Section 4.5, arXiv:1907.07271).

[IMAGE: Line chart, tutoring hours 0 to 12 against exam score. A grey line shows the population interventional mean. Three coloured lines pass through the observed point (4, 120) with total slopes 7.5, 9.5 and 6.5 points per hour, reaching 165, 177 and 159 at X=10. Caption: "Every line fits the student's data and every experiment. Only an assumption about unit-level noise picks one."]

Where It Breaks

The measure-zero set is where practitioners live

"Almost never" is relative to a smooth measure over all SCMs. Real analyses restrict the class (linear Gaussian with known ordering, additive noise, monotone response) to measure-zero subsets where higher layers can follow from lower ones, importing the missing layer through the model class. The failure is doing it silently, as any package printing unit-level counterfactuals from an additive-noise fit does.

Cross-world assumptions are untestable by construction

Layer-3 answers rest on joint distributions over outcomes never observed together. Part 1 shows every experiment agreeing while \(PN\) is 0 in one model and 1 in the other. A personalised "what would have happened" number without bounds or an explicit monotonicity or rank argument is an assumption with a decimal point. The semantics also presuppose an ordering of mechanisms, which supply-and-demand loops lack; Imbens presses this when The Book of Why asks readers not to "lose too much sleep" over two-way arrows in Wright's diagram.

Benchmarks that mix recall with inference

GPT-4 reproduced 61% of the long-public Tübingen cells, though Kıcıman et al. note accuracy exceeds what memorisation explains. Zečević and colleagues formalise the worry as a meta SCM, whose variables are causal facts about other models, and conjecture that LLM successes reflect correlations among such facts in training text, making the models "causal parrots" (Zečević et al., 2023, TMLR, arXiv:2308.13067). CLadder and Corr2Cause measure computation in natural language, so parsing, arithmetic and inference errors are conflated.

The theorem is misread in both directions

One misreading says the CHT proves LLMs cannot answer causal questions. It does not: text is not Layer-1 data about the variables asked about but people reporting experiments and mechanisms, higher-layer information in linguistic form. The opposite misreading takes correct answers as evidence the model inferred causation, exactly what the CHT forbids concluding from correlations. Kıcıman et al. and Zečević et al. both locate LLM causal competence in encoded knowledge; they differ on whether that makes a useful front end or a misleading echo.

Alternative Designs

Framework	Primitive	Rung reached	Key advantage	Key limitation	Best when
SCMs and DAGs	Functions plus noise	3	One object for all rungs; complete identification	Unit-level answers need functional form	Many variables; is the query answerable?
Potential outcomes	\(Y_i(0), Y_i(1)\) per unit	2 routinely, 3 with explicit assumptions	Assignment mechanism and monotonicity are natural	Ad hoc identification in large systems	Few variables, strong design: RCT, IV, RD, DiD
Partial identification	Sets of consistent models	2 and 3 as intervals	Honest about what data cannot settle	Intervals may be too wide to act on	Attribution, liability, safety
Causal discovery	Independences plus faithfulness	Graph fragments	Structure from Layer-1 data	Faithfulness is a strong assumption	Exploration before experiments
Neural causal models	SCM with neural functions	2 and 3 when identified	Flexible mechanisms plus identification theory	CHT still binds	High-dimensional mechanisms, known structure
LLM as knowledge source	Text-encoded causal claims	What the authors knew	Fast candidate graphs from names	Recall, not inference; unpredictable errors	Drafting graphs for expert and data review

The PO-versus-DAG dispute is sharper than a table. Imbens argues economists favour potential outcomes because monotonicity and other shape restrictions central to instrumental variables are "not easily incorporated in the DAGs and the attendant do-calculus," and that the DAG literature has few "substantive empirical examples." Pearl's January 2020 reply counters that SCMs contain \(Y_x\) as a derived quantity, so monotonicity is written identically, and that good versus bad controls are "not trivial to articulate without the help of graphs" (Pearl, 2020, blog reply to Imbens). Fairly read, the frameworks are interconvertible for most queries and differ in which assumptions they make visible. In Part 2, SCM notation makes \(U_Y\) look like one thing; potential-outcomes notation makes you write \(U_Y(4)\) and \(U_Y(10)\) and ask whether they are equal.

How It Is Used in Practice

Identify before estimating. DoWhy and dagitty put identification before estimation, so an unidentifiable query stops an analysis before it prints a confident number.

Rung 2 for experiments, rung 3 for attribution. Conditional average effects on pre-treatment covariates are Layer 2 and estimable from RCTs. "Would this churned customer have stayed without the price change?" is Layer 3. The effect of treatment on the treated is formally counterfactual yet identifiable whenever a backdoor adjustment set exists. Individual-level uplift scores lean on rank assumptions like Part 2's.

Legal and safety attribution. \(PN\) formalises "but-for" causation; when the Tian-Pearl bounds are wide, report the interval and add observational data on who chose treatment.

LLMs in the workflow. The use that survives scrutiny as of September 2026 is LLM-as-domain-expert: propose candidate edges from variable names, then test them against data and expert review. Kıcıman et al. frame the opportunity this way while noting the models "make simple, unpredictable mistakes on certain inputs." As the inference engine itself, LLMs are where CLadder, Corr2Cause and Caliper find weakness; call an identification library instead.

[IMAGE: Pipeline. Box 1 "LLM proposes edges from variable names" (amber, labelled recall). Box 2 "Independence tests on data" (blue). Box 3 "Expert review" (slate). Box 4 "Identification library" (purple, labelled computation). Box 5 "Estimate or bound" (teal). A red dashed bypass from box 1 to box 5 is struck through. Caption: "Let the model supply candidate assumptions; never let it skip identification."]

Insights Worth Remembering

The ladder is about information, not intelligence. A perfect learner with a trillion observational rows still cannot recover \(P(y \mid do(x))\) in general; the theorem binds neural networks exactly as it binds regressions.
Every climb is paid in assumptions. The graph buys rung two; functional form or monotonicity buys rung three. A method that seems to climb free smuggled the assumption in through its model class.
Completeness turns "we could not identify it" into "it cannot be identified." A failed identification is a result: run an experiment or elicit an assumption; more observational data will not help.
Additive noise quietly makes every unit average. With a common slope, individual counterfactual effects equal the average effect by construction.
Experiments do not settle attribution. Part 1's models agree on every randomised result and disagree completely about whether treatment caused the outcome for the unit in front of you.
Text is not Layer-1 data. The CHT does not stop a language model knowing that smoking causes cancer, because that sentence reports higher-rung evidence. It stops us crediting that knowledge to correlational pattern extraction.

Open Questions

Do reasoning models execute the inference engine or imitate it? Measured: GPT-4 was nearly flat across CLadder rungs, and the Caliper preprint reports mean drops of 6.7 to 7.6 points on CLadder's interventional rung, and up to 29.6 on CRASS, when variable names are anonymised. Unknown: whether reasoning-focused training closes the gap on anonymised items, which needs benchmarks released after training cut-offs.

Can shape restrictions become first-class in graphical identification? Imbens wrote in 2020 that monotonicity, raised against Pearl's 1995 paper, had not been addressed 25 years later. A complete identification algorithm for graphs plus monotonicity appears, from the literature reviewed here, still open.

How much information must assumptions carry? A May 2026 preprint constructs binary SCMs where the observational distribution takes constant bits but the single-variable interventional answers need on the order of \(n^2\) bits, and counterfactuals can need order \(n\) more even given every interventional answer (Emadi, 2026, arXiv:2605.02177). Whether real systems approach these worst cases is unmeasured.

Is the meta-SCM hypothesis testable? Zečević et al. offer it as a conjecture with limited evidence. A decisive test would control how often causal relations appear in training text, including true ones rarely written down; it has not been run at scale.

Sources and Further Reading

Bareinboim, E., Correa, J. D., Ibeling, D., & Icard, T. (2022). "On Pearl's Hierarchy and the Foundations of Causal Inference." In Probabilistic and Causal Inference: The Works of Judea Pearl, ACM Books, 507-556. Technical Report R-60, 2020. PDF
Pearl, J. (1995). "Causal Diagrams for Empirical Research." Biometrika, 82(4), 669-688. doi:10.1093/biomet/82.4.669
Shpitser, I., & Pearl, J. (2006). "Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models." AAAI 2006, 1219-1226. eScholarship
Huang, Y., & Valtorta, M. (2006). "Pearl's Calculus of Intervention Is Complete." UAI 2006, 217-224. arXiv:1206.6831
Shpitser, I., & Pearl, J. (2008). "Complete Identification Methods for the Causal Hierarchy." JMLR, 9, 1941-1979. JMLR
Balke, A., & Pearl, J. (1994). "Counterfactual Probabilities: Computational Methods, Bounds and Applications." UAI 1994. arXiv:1302.6784
Tian, J., & Pearl, J. (2000). "Probabilities of Causation: Bounds and Identification." Annals of Mathematics and Artificial Intelligence, 28, 287-313. doi:10.1023/A:1018912507879, UCLA R-271-A
Pearl, J. (2019). "The Seven Tools of Causal Inference, with Reflections on Machine Learning." Communications of the ACM, 62(3), 54-60. doi:10.1145/3241036, author version R-481
Pearl, J., & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.
Imbens, G. W. (2020). "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics." Journal of Economic Literature, 58(4), 1129-1179. AEA, arXiv:1907.07271
Pearl, J. (2020, January 29). "On Imbens's Comparison of Two Approaches to Empirical Economics." Causality blog
Rubin, D. B. (1974). "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies." Journal of Educational Psychology, 66(5), 688-701. doi:10.1037/h0037350
Wright, S. (1934). "The Method of Path Coefficients." Annals of Mathematical Statistics, 5(3), 161-215. doi:10.1214/aoms/1177732676
Ibeling, D., & Icard, T. (2021). "A Topological Perspective on Causal Inference." NeurIPS 2021. arXiv:2107.08558
Xia, K., Lee, K.-Z., Bengio, Y., & Bareinboim, E. (2021). "The Causal-Neural Connection: Expressiveness, Learnability, and Inference." NeurIPS 2021. arXiv:2107.00793
Kıcıman, E., Ness, R., Sharma, A., & Tan, C. (2023). "Causal Reasoning and Large Language Models: Opening a New Frontier for Causality." arXiv:2305.00050
Zečević, M., Willig, M., Dhami, D. S., & Kersting, K. (2023). "Causal Parrots: Large Language Models May Talk Causality But Are Not Causal." TMLR. arXiv:2308.13067
Jin, Z., Chen, Y., Leeb, F., Gresele, L., et al. (2023). "CLadder: Assessing Causal Reasoning in Language Models." NeurIPS 2023. arXiv:2312.04350

Further reading: Jin, Z., Liu, J., et al. (2024), "Can Large Language Models Infer Causation from Correlation?", ICLR 2024, arXiv:2306.05836; Yu, Z., & Zhou, S. (2026), "Caliper: Probing Lexical Anchors versus Causal Structure in LLMs," preprint, arXiv:2606.04915; Emadi, S. M. (2026), "The Causal Description Gap," preprint, arXiv:2605.02177.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.