What a Feature Attribution Can and Cannot Tell You: SHAP, LIME and the Explanation Gap
Add a column the model never reads and SHAP can hand it more than a quarter of the credit for a decision. That is not a library bug: a Shapley attribution answers a question you chose, often without noticing, and a 2024 PNAS result shows that for rich enough models no complete, linear attribution can reliably answer the counterfactual questions people read into it.
In 2021 a Nature study of cancer-trial eligibility criteria read its conclusions off Shapley values: criteria with values close to zero "had no effect on the hazard ratio of the overall survival." Bilodeau, Jaques, Koh and Kim later used that sentence to motivate a proof that the inference is not licensed. For moderately rich model classes, which neural networks satisfy, a zero attribution from any complete and linear method does not imply the output is insensitive to the feature, and a positive one does not imply that increasing the feature increases the output (Bilodeau et al., 2024, PNAS 121(2), arXiv:2212.11870).
SHAP, LIME and integrated gradients do what their papers say, but each silently answers a specific question (which baseline, which distribution, which columns are players) while the reader asks another: what would happen if this feature were different? The worked example below makes the gap arithmetic: one applicant, one model, three defensible Shapley attributions, one of which hands 27 percent of the credit to a column the model never reads.
Why this matters: Feature attributions are default evidence in model reviews, fairness audits, lending adverse-action reasons and EU AI Act explanations. Knowing which question an attribution answers, and which it provably cannot, separates evidence that survives scrutiny from evidence an adversary, or an honest correlation, can quietly falsify.
TL;DR
- "The SHAP value" is not one number. Baseline, interventional and observational Shapley values obey the same axioms yet disagree; in the example below, income gets 14, 8 or 3.83 points.
- Uniqueness is relative to a game you invented. Shapley's axioms fix how to split a payout once the value function is chosen, and are silent on which one is right.
- Observational Shapley credits columns the model never uses. A perfect proxy takes as much credit as the feature it copies; interventional Shapley gives it zero but queries impossible inputs.
- Those impossible inputs are an attack surface. Slack et al. disguised a race-only classifier so race stopped being the top LIME feature on 100 percent of held-out COMPAS points, and the top SHAP feature on 84 percent.
- Completeness plus linearity cannot certify counterfactual behaviour. Bilodeau et al. prove such methods can fail to beat random guessing at recourse and spurious-feature detection; SHAP and integrated gradients were near random in almost all their experiments on eight datasets.
- Explainers disagree in routine use. 84 percent of 25 interviewed practitioners had hit disagreement; 86 percent of study participants resolved it by ad hoc heuristics or were unsure.
- Generate hypotheses with attributions, test them with direct model queries. That is Bilodeau et al.'s own recommendation.
At a Glance
flowchart LR
X["One prediction f(x)"] --> Q{"Hidden choices"}
Q --> V1["Value function"]
Q --> V2["Baseline or background"]
Q --> V3["Which features are players"]
V1 --> A["Attribution vector"]
V2 --> A
V3 --> A
A --> R["Reader asks a counterfactual"]
R --> G["Explanation gap"]
G --> T["Test by querying the model"]
classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
class X blue
class Q,V1,V2,V3 amber
class A purple
class R teal
class G rose
class T emeraldSeventy Years of Splitting a Payout
Lloyd Shapley's "A Value for n-Person Games" (1953) asked: if players jointly earn \(v(N)\), and we know what every sub-coalition would earn alone, what is each player's fair share? He derived a unique answer from three axioms, which modern treatments unpack into four: efficiency, symmetry, dummy and additivity (Shapley, 1953, Annals of Mathematics Studies 28). Aumann and Shapley extended it in 1974 to a continuum of players, the construction integrated gradients would rediscover.
Štrumbelj and Kononenko made it local and model-agnostic in 2014, but mainstream adoption came from two papers a year apart. LIME (Ribeiro, Singh & Guestrin, 2016, KDD, arXiv:1602.04938) fitted a sparse linear model to a black box around one input. Its best-known demo was a deliberately bad husky-versus-wolf classifier trained on 20 images where every wolf photo had snow: before explanations, 10 of 27 graduate students trusted it and 12 named snow as a possible feature; afterwards, 3 and 25. SHAP (Lundberg & Lee, 2017, NeurIPS, arXiv:1705.07874) placed LIME, DeepLIFT and Shapley sampling in one class of additive feature attributions in which only Shapley values satisfy local accuracy, missingness and consistency. Integrated gradients gave deep networks an axiomatic path method (Sundararajan, Taly & Yan, 2017, ICML, arXiv:1703.01365), and TreeSHAP made exact tree values fast, merged into XGBoost and LightGBM (Lundberg, Erion & Lee, 2018, arXiv:1802.03888). The critique arrived almost as fast.
timeline
title From Coalition Games to Impossibility Theorems
1953 : Shapley derives a unique fair split of a coalition payout
1974 : Aumann and Shapley extend the value to non-atomic games
2016 : LIME fits sparse local surrogates to any classifier
2017 : SHAP unifies additive attributions under Shapley axioms
: Integrated gradients brings path attribution to deep nets
2018 : TreeSHAP computes exact values in polynomial time
: Sanity checks show some saliency maps ignore the model
2020 : Slack et al. fool LIME and SHAP with a scaffolded classifier
: Janzing, Chen, Kumar and Sundararajan argue over the value function
2024 : Bilodeau et al. publish impossibility theorems in PNAS[IMAGE: Two panels. Left, 1953: a lattice of all 8 sub-coalitions of three players with their payouts. Right: the same lattice with income, debt and postcode as players, every node but the full coalition marked "?". Caption: "A model defines the worth of the full coalition only; the rest is invented."]
How Feature Attribution Actually Works
The game you have to invent
A cooperative game needs a value function \(v(S)\) for every subset \(S\) of the \(M\) features, but a model \(f\) only defines the grand coalition, \(v(N) = f(x)\). Everything else is constructed, and the construction is where every disagreement in this literature lives. Given \(v\), feature \(i\)'s Shapley attribution is its weighted average marginal contribution:
The weight is the fraction of the \(M!\) orderings in which exactly \(S\) arrives before \(i\): reveal features in random order and credit each with the change in worth on arrival. For \(M = 3\) the weights are \(1/3\) for the empty set, \(1/6\) for each singleton and \(1/3\) for each pair.
Three constructions of \(v\) dominate. Baseline Shapley fills absent features from a reference input \(x'\); interventional Shapley averages over a background distribution drawn independently of the present features; observational (conditional) Shapley conditions on what the present features reveal:
The first two ask how the model behaves when you overwrite features; the third asks what it is expected to output when you learn some. Under independence the last two coincide; under correlation they are different games. Sundararajan and Najmi put it bluntly: the operationalisations "give very different results, rendering the uniqueness result meaningless" (Sundararajan & Najmi, 2020, ICML, PMLR 119:9269-9278).
What the axioms buy, and what they do not
Efficiency forces \(\sum_i \phi_i = v(N) - v(\varnothing)\). Symmetry gives equal credit to features that contribute identically to every coalition. Dummy gives zero to a feature whose arrival never changes \(v\). Additivity makes attributions of a sum of games add, so an ensemble can be explained tree by tree.
Every one is a statement about \(v\), not about \(f\). Under observational Shapley, a feature the model ignores can still change \(v\) by revealing information about features it uses, so dummy does not protect it. Kumar, Venkatasubramanian, Scheidegger and Friedler add that additivity, which game theorists called "not nearly so innocent as the other two", costs something on non-additive models: for a product of independent zero-centred features, \(f(x) = \prod_j x_j\), every feature receives \(f(x)/d\) whatever its own value (Kumar et al., 2020, ICML, PMLR 119:5491-5500, arXiv:2002.11097). The axioms share interactions evenly, whatever the reader thinks "importance" means.
The causal argument over the value function
The SHAP paper defined \(v\) as a conditional expectation, then approximated it by sampling absent features independently. Janzing, Minorics and Blöbaum argued the approximation was right and the definition wrong: removing a model input is an intervention in Pearl's sense, so "unconditional rather than conditional expectations provide the right notion of dropping features", and attempts to "improve" SHAP by estimating the conditional were conceptually flawed (Janzing, Minorics & Blöbaum, 2020, AISTATS, PMLR 108:2907-2916, arXiv:1910.13413).
The improvements they meant were careful: Aas, Jullum and Løland estimated the conditional with Gaussian, copula and empirical methods (arXiv:1903.10464); Frye, Rowat and Feige averaged only over orderings consistent with a causal graph (arXiv:1910.06358).
Chen, Janizek, Lundberg and Lee, two of them SHAP's authors, reframed the fight as a choice of purpose. For a linear model \(f(x) = \beta^\top x + b\) with independent background, the interventional attribution has a closed form,
which depends only on the model's own coefficient: true to the model. The observational attribution spreads \(\beta_i\)'s effect across every feature correlated with \(x_i\): true to the data. A bank explaining a denied loan wants the first; a biologist asking which genes carry information about an outcome may want the second (Chen et al., 2020, arXiv:2006.16234). Kumar et al. call the fork a catch-22: conditional methods need many conditional distributions and credit features with no interventional effect; interventional methods leave the data manifold. Both camps agree on the facts, and disagree on whether two answers is a feature or a fatal flaw.
[IMAGE: Scatter of income against postcode wealth, real data a thin diagonal band. Blue observational samples hug the band; red interventional samples fill the square. Caption: "Interventional Shapley questions the model about points no real applicant occupies."]
LIME and KernelSHAP: one regression, two kernels
LIME defines no value function. It samples perturbations \(z\) near \(x\), weights them by a proximity kernel \(\pi_x\), and fits an interpretable \(g\):
The reference tabular code uses \(\pi_x(z) = \sqrt{\exp(-D(x,z)^2/\sigma^2)}\) with default \(\sigma = 0.75\sqrt{p}\) for \(p\) columns and 5,000 samples (lime source). Nothing forces the weights to sum to anything, and the answer moves with \(\sigma\). Lundberg and Lee showed that one kernel over binary coalition vectors \(z'\), with no regulariser, turns the same regression into the Shapley value:
It is infinite at the empty and full coalitions, enforcing efficiency. KernelSHAP is LIME with an axiom-derived kernel, and inherits LIME's sampling: shap defaults to \(2M + 2048\) coalition evaluations, each averaged over a background set (shap _kernel.py).
Integrated gradients and the cost of exactness
For differentiable models, integrated gradients attributes along a straight path from a baseline \(x'\):
It satisfies completeness (\(\sum_i \mathrm{IG}_i = f(x) - f(x')\)); its authors found 20 to 300 Riemann steps get within 5 percent. As the Aumann-Shapley value of a baseline game it inherits the baseline question whole: a black image, a blurred image and the training mean are three different games.
Exactness is expensive: \(2^{M-1}\) subsets per feature, over a million coalitions at \(M = 20\). Van den Broeck and colleagues showed the difficulty is intrinsic: under a fully factorised distribution SHAP is as hard as the model's expectation, already intractable for logistic regression, and over the empirical distribution it is #P-hard (Van den Broeck et al., 2021, AAAI, arXiv:2009.08634). TreeSHAP cuts trees from \(O(TL2^M)\) to \(O(TLD^2)\) for \(T\) trees, \(L\) leaves and depth \(D\). Elsewhere the number you read is a Monte Carlo estimate of one member of a family of games.
Seeing the Gap
The first diagram maps questions to constructions, and where each breaks.
flowchart TB
Q["What do you want to know?"] --> M1["How this model uses inputs"]
Q --> M2["What the data says"]
Q --> M3["What this person should change"]
M1 --> I["Interventional or baseline Shapley"]
M2 --> OB["Observational Shapley"]
M3 --> C["Direct counterfactual queries"]
I --> R1["Risk: off-manifold, spoofable"]
OB --> R2["Risk: unused proxies get credit"]
C --> R3["Risk: must define feasible changes"]
classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
class Q blue
class M1,M2,M3 teal
class I,OB,C purple
class R1,R2,R3 roseThe second traces the attack Slack and colleagues built on the interventional route: a scaffold sends realistic inputs to a biased model and synthetic perturbations to an innocuous one (Slack et al., 2020, Fooling LIME and SHAP, AIES, arXiv:1911.02508).
sequenceDiagram
participant A as Auditor
participant E as SHAP or LIME
participant S as Scaffold
participant B as Biased model
participant D as Decoy model
A->>S: Real applicant x
S->>B: In-distribution, route to biased model
B-->>A: Decision driven by race
A->>E: Explain the decision on x
E->>S: Thousands of perturbed samples
S->>D: Flagged as synthetic, route to decoy
D-->>E: Outputs depend on an uncorrelated feature
E-->>A: Top feature is the uncorrelated one
Note over A,E: The explanation describes the decoyBy the Numbers
| Quantity | Value | What it shows | Source |
|---|---|---|---|
| Exact SHAP over the empirical distribution | #P-hard | Hardness is intrinsic | Van den Broeck et al., 2021 |
| KernelSHAP default budget | \(2M + 2048\) evaluations | Default output is a sampled estimate | shap source |
| Husky vs wolf, subjects naming snow | 12 of 27 before, 25 of 27 after | Explanations can expose a spurious feature | Ribeiro et al., 2016 |
| Sensitive feature displaced from LIME top rank | 100% COMPAS and Communities and Crime | Perturbation explainers can be spoofed | Slack et al., 2020 |
| Displaced from SHAP top rank, one decoy feature | 84% COMPAS, 100% C&C, 85% German Credit | Efficiency resists, but not much | Slack et al., 2020 |
| Scaffold agreement with biased model, SHAP attack | 91% COMPAS, 80% German, 75% C&C | Fooling SHAP costs some fidelity | Slack et al., 2020 |
| Explainer disagreement | 84% of 25 interviewees met it; 86% of users resolved it ad hoc or unsure | No principled resolution | Krishna et al., 2024 |
| ResNet-50, 90% of ImageNet inputs replaced, retrained | 63.53% vs 76.68% clean | Deletion tests confound distribution shift | Hooker et al., 2019 |
| SHAP and IG on recourse and spurious-feature tasks | Near random in almost all experiments, 8 datasets | Complete, linear attributions miss local behaviour | Bilodeau et al., 2024 |
Sources: papers cited in the text (Ribeiro et al. Table 2; Slack et al. and Bilodeau et al. Section 4); shap source as of September 2026. The sensitive feature is race for COMPAS and Communities and Crime, gender for German Credit.
A Concrete Example: Three Honest Answers for One Applicant
A lender's model reads binary features \(x_1\) = high income, \(x_2\) = low debt, \(x_3\) = affluent postcode, with an interaction, and never reads the postcode:
In the population, income and debt are independent with \(P(x_1 = 1) = P(x_2 = 1) = 0.5\), and postcode is a perfect proxy for income, \(x_3 = x_1\). Explain applicant \(x = (1, 1, 1)\), who scores \(f(x) = 24\).
Step 1: the reference. \(\mathbb{E}[f] = 10(0.5) + 6(0.5) + 8(0.25) = 10\), so interventional and observational attributions must each sum to \(24 - 10 = 14\).
Step 2: the interventional game. Fill absent features from their marginals. Postcode never enters \(f\), so adding it never changes \(v\):
| Coalition \(S\) | \(v_{\text{int}}(S)\) | Arithmetic |
|---|---|---|
| \(\varnothing\), \(\{3\}\) | 10 | population mean |
| \(\{1\}\), \(\{1,3\}\) | 17 | \(10 + 6(0.5) + 8(0.5)\) |
| \(\{2\}\), \(\{2,3\}\) | 15 | \(10(0.5) + 6 + 8(0.5)\) |
| \(\{1,2\}\), \(\{1,2,3\}\) | 24 | full prediction |
Step 3: weigh the marginals. Income adds 7 to \(\varnothing\) (weight \(1/3\)), 9 to \(\{2\}\) (\(1/6\)), 7 to \(\{3\}\) (\(1/6\)) and 9 to \(\{2,3\}\) (\(1/3\)):
Likewise \(\phi_2^{\text{int}} = \tfrac{5}{3} + \tfrac{7}{6} + \tfrac{5}{6} + \tfrac{7}{3} = 6\) and \(\phi_3^{\text{int}} = 0\); sum 14. Main effects are 5 for income and 3 for debt; the interaction sits \(8 - 2 = 6\) above its average, and the axioms split it 3 and 3.
Step 4: the observational game. Now condition. Knowing the postcode reveals income, so \(v_{\text{obs}}(\{3\}) = \mathbb{E}[f \mid x_1 = 1] = 17\) and \(v_{\text{obs}}(\{2,3\}) = 24\). Every other entry matches the table.
Step 5: recompute. Income still adds 7 to \(\varnothing\) and 9 to \(\{2\}\), but 0 to \(\{3\}\) and 0 to \(\{2,3\}\): the postcode already told the game what income would say.
By symmetry \(\phi_3^{\text{obs}} = 23/6\), and debt gets \(\tfrac{5}{3} + \tfrac{7}{6} + \tfrac{7}{6} + \tfrac{7}{3} = \tfrac{38}{6} \approx 6.33\). Sum: 14. The postcode, never read by the model, carries \(3.83/14 = 27\) percent of the explanation.
Step 6: the baseline game. Against the all-zero applicant, \(v(\varnothing) = 0\), \(v(\{1\}) = 10\), \(v(\{2\}) = 6\), \(v(\{1,2\}) = 24\). Income gets \(\tfrac{1}{2}(10 + 18) = 14\), debt \(\tfrac{1}{2}(6 + 14) = 10\), postcode 0; total 24. Integrated gradients on the continuous extension from the same baseline agrees: \(\int_0^1 (10 + 8\alpha)\,d\alpha = 14\) and \(\int_0^1 (6 + 8\alpha)\,d\alpha = 10\).
Step 7: drop the proxy. Leave postcode out and the observational game on the independent pair \(\{x_1, x_2\}\) equals the interventional one: income is back to 8. A redundant column moved income from 8 to 3.83 without changing one prediction, the redundancy effect Kumar et al. describe.
| Method | Income | Debt | Postcode | Sum | Question answered |
|---|---|---|---|---|---|
| Baseline Shapley, zero reference | 14 | 10 | 0 | 24 | Credit versus an applicant with nothing |
| Interventional Shapley | 8 | 6 | 0 | 14 | How this model uses inputs, versus average |
| Observational Shapley | 3.83 | 6.33 | 3.83 | 14 | What each fact reveals about the score |
No row is an error. Reviewers told "income contributed 14 points" and "3.83 points" were both told the truth about different games.
[IMAGE: Grouped bar chart of the table above: three method groups, bars for income (blue), debt (purple) and postcode (amber). The postcode bar appears only in the observational group. Caption: "One applicant, one model, three defensible attributions."]
Where It Breaks
The off-manifold query is an attack surface
The interventional game evaluates \(f\) on stitched rows, such as a high-income applicant in a poor postcode, that may never occur, and nothing constrains the model there. Slack et al. trained a detector to separate real rows from perturbations and used it to pick the biased or decoy model. SHAP resisted slightly better than LIME because efficiency spreads credit when no single decoy can absorb it: with two decoys, race was displaced in only 67 and 71 percent of points. The attacker pays in fidelity, since SHAP scaffolds matched the biased model on 75 to 91 percent of held-out rows. Even without an attacker, behaviour in empty regions is unregularised, and attributions that average over them report it.
[IMAGE: PCA projection of COMPAS rows as a tight blue cloud, LIME perturbations as a diffuse red cloud beyond it, and the detector's dashed boundary between them, after Figure 1 of Slack et al. Caption: "The scaffold only has to tell real rows from synthetic ones."]
The attribution does not determine the behaviour
Bilodeau et al. frame the reader's question as a hypothesis test: near \(x\), is \(f\) increasing in feature \(j\), or sensitive to it? For complete and linear methods over rich model classes, "there are uncountably many pairs of models that share a feature attribution yet have arbitrarily different counterfactual model behaviour." Completeness forces attributions to sum to a global gap, which is not a local slope, and reliance on a baseline lets behaviour far from \(x\) contaminate the attribution however small the neighbourhood of interest. Hence gradients, SmoothGrad and LIME beat SHAP and IG on their simple tabular data, while on images every method was near random. A positive SHAP value for credit score does not tell a denied applicant that raising it will help; that is a job for counterfactual explanations and algorithmic recourse.
Interactions and the player set are decided by fiat
The Shapley value cannot say "these two only matter together"; it splits joint effects equally, as income and debt each took 3 of the 6 interaction points above. The player set is an equally silent decision. Under observational Shapley a copied feature shares its credit with the copy; under interventional Shapley an engineered \(x_3 = x_1 x_2\) is overwritten independently of its parents. Kumar et al. note that two fairness audits of the same function, differing only in whether a sensitive attribute's perfect proxy counts as a player, "would come out with quantitatively different results."
Saliency that ignores the model
Adebayo et al. randomised a trained network's weights layer by layer. Gradients and GradCAM passed; Guided BackProp and Guided GradCAM were invariant to higher-layer parameters, their maps strikingly like an edge detector's (Adebayo et al., 2018, NeurIPS, arXiv:1810.03292). Deletion tests are confounded too: a ResNet-50 retrained with 90 percent of ImageNet inputs randomly replaced still reached 63.53 percent, and under retraining many popular methods did no better than random (Hooker et al., 2019, NeurIPS, arXiv:1806.10758).
Disagreement and noise
Practitioners resolve conflicting explanations by heuristics such as preferring a familiar method (Krishna et al., 2024, TMLR, arXiv:2202.01602). KernelSHAP and LIME are Monte Carlo estimates too: mid-ranked features reorder between seeds, and a k-means background (Slack et al. used 10 clusters) shifts \(\mathbb{E}[f]\) itself.
Alternative Designs
| Design | How it works | Key advantage | Key limitation | Best when |
|---|---|---|---|---|
| Interventional SHAP | Absent features from marginal background | True to the model | Queries impossible inputs; spoofable | Debugging a model's use of weakly dependent inputs |
| Observational SHAP | Condition on present features | Stays on the data manifold | Credits unused proxies | Asking what the data says about the output |
| Baseline Shapley and IG | Fixed reference, straight path | Deterministic | Only as meaningful as the baseline | A natural reference exists, such as zero dose |
| Asymmetric Shapley | Orderings restricted by a causal graph | Causal ancestors take credit | Sensitive to causal assumptions | A defensible causal graph exists |
| LIME | Sparse surrogate near \(x\) | Cheap, any model | Arbitrary width; spoofable | Quick hypotheses on text or images |
| Direct counterfactual queries | Evaluate \(f\) on realistic edits of \(x\) | Answers recourse directly | Needs feasible edits defined | Adverse action, spurious-feature tests |
| Interpretable model | GAM, scorecard or sparse rules | The explanation is the model | Possible accuracy cost on raw pixels and text | High-stakes tabular decisions |
The last row is Rudin's position: post-hoc explanations "cannot have perfect fidelity" to the model, and on structured data with naturally meaningful features a black box is often not needed for top accuracy (Rudin, 2019, Nature Machine Intelligence 1:206-215, arXiv:1811.10154). The reply is that no scorecard reads a chest X-ray; the error is importing that defence into a credit decision where a monotone GAM would do. Bilodeau et al. sit between: define the end task, then answer it with repeated model evaluations, which "can outperform many other complex feature attribution methods."
[IMAGE: A 2x2 decision matrix. Axes: can the stakeholder's question be written as a specific counterfactual; is an interpretable model competitive on this data. Quadrants: "interpretable model", "interpretable model plus counterfactual checks", "direct queries on the black box", "attributions for hypotheses only". Caption: "Start from the question and the data, not from the installed explainer."]
How It Is Used in Practice
Tree models. TreeSHAP made exact attributions routine in tabular pipelines (Lundberg et al., 2020, Nature Machine Intelligence 2:56-67). Its default path-dependent mode approximates a conditional expectation from node cover counts, the interventional mode uses an explicit background, and on correlated features they can diverge; record which ran.
Adverse-action reasons in US lending. CFPB Circular 2022-03 said a creditor cannot excuse missing adverse-action reasons because its technology "is too complicated or opaque to understand" (CFPB, May 2022). It was among 67 guidance documents withdrawn on 12 May 2025 (Federal Register 2025-08286); the statute and Regulation B remain. SHAP-derived reason codes answer "what did the model use", not "what should you change".
The EU AI Act. Article 86 of Regulation (EU) 2024/1689 gives people affected by certain high-risk decisions a right to "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken" (Article 86). A SHAP chart is one input; see explaining a decision to the person affected.
Documentation. An attribution is reproducible only if its game is written down: library version, value function, background, output scale, grouping, samples, seeds and stability. That record belongs with documentation that survives an audit; the concepts on LIME and SHAP and TreeSHAP cover the methods themselves. A workable review treats each attribution as a hypothesis with a status:
stateDiagram-v2
[*] --> Computed
Computed --> Configured: value function and background recorded
Configured --> Stable: ranks hold across seeds
Configured --> Discarded: ranks reorder
Stable --> Hypothesis: feature flagged as driver or spurious
Hypothesis --> Confirmed: direct queries agree
Hypothesis --> Refuted: direct queries disagree
Refuted --> Computed: revise and rerun
Confirmed --> Reported
Reported --> [*]
Discarded --> [*][IMAGE: An explanation record as a form: explainer version, value function, background (n=100, k-means), output scale, grouped features, seeds, rank correlation across reruns, and a "confirmed by direct query" box per top feature. Caption: "An attribution without its configuration is not evidence."]
Insights Worth Remembering
-
Uniqueness is conditional on the game. The Shapley value is the only allocation satisfying its axioms for a given value function. Choosing that function has at least three defensible answers.
-
Dummy protects features from the value function, not from the model. An unread column took 27 percent of observational credit above without violating any axiom.
-
"True to the model" and "true to the data" are different deliverables. Reporting one when the reader wants the other is the most common silent error.
-
Completeness is bookkeeping, not behaviour. Attributions summing to \(f(x) - \mathbb{E}[f]\) allocate a global gap, and Bilodeau et al. show that constraint helps decouple them from local counterfactual behaviour.
-
The cheapest reliable explanation is often a direct query. If the question is "would changing this feature change this decision", evaluate the model on realistic changed inputs.
Open Questions
Can a value function be both on-manifold and true to the model? Asymmetric Shapley values help when a causal graph is known, but Kumar et al. flag their sensitivity to how much causal knowledge is supplied. Without a trusted graph, no middle ground is established.
Can stakeholder questions be compiled into sufficient queries? Bilodeau et al. show repeated model evaluation works once the task is defined, but no general recipe turns a question into a small sufficient query set, least of all for images.
How detectable are scaffolding attacks in real audits? Slack et al. show the attack depends on the detector's accuracy. Whether on-manifold samplers or fidelity checks defeat adaptive versions is barely measured; claims either way are speculative.
What will regulators accept as a reason? Article 86 and US adverse-action rules specify no method. Whether supervisors accept SHAP rankings, and what faithfulness evidence they demand, has little public enforcement record as of September 2026.
Sources and Further Reading
- Shapley, L. S. (1953). "A Value for n-Person Games." Contributions to the Theory of Games II, Annals of Mathematics Studies 28, 307-317. doi:10.1515/9781400881970-018
- Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "'Why Should I Trust You?': Explaining the Predictions of Any Classifier." KDD. arXiv:1602.04938
- Lundberg, S. M., & Lee, S.-I. (2017). "A Unified Approach to Interpreting Model Predictions." NeurIPS. arXiv:1705.07874
- Sundararajan, M., Taly, A., & Yan, Q. (2017). "Axiomatic Attribution for Deep Networks." ICML. arXiv:1703.01365
- Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018). "Consistent Individualized Feature Attribution for Tree Ensembles." arXiv:1802.03888
- Janzing, D., Minorics, L., & Blöbaum, P. (2020). "Feature Relevance Quantification in Explainable AI: A Causal Problem." AISTATS, PMLR 108. arXiv:1910.13413
- Chen, H., Janizek, J. D., Lundberg, S., & Lee, S.-I. (2020). "True to the Model or True to the Data?" arXiv:2006.16234
- Sundararajan, M., & Najmi, A. (2020). "The Many Shapley Values for Model Explanation." ICML, PMLR 119. arXiv:1908.08474
- Kumar, I. E., et al. (2020). "Problems with Shapley-value-based Explanations as Feature Importance Measures." ICML, PMLR 119. arXiv:2002.11097
- Slack, D., et al. (2020). "Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods." AIES. arXiv:1911.02508
- Bilodeau, B., Jaques, N., Koh, P. W., & Kim, B. (2024). "Impossibility Theorems for Feature Attribution." PNAS 121(2). doi:10.1073/pnas.2304406120
- Van den Broeck, G., et al. (2021). "On the Tractability of SHAP Explanations." AAAI. arXiv:2009.08634
- Adebayo, J., et al. (2018). "Sanity Checks for Saliency Maps." NeurIPS. arXiv:1810.03292
- Hooker, S., et al. (2019). "A Benchmark for Interpretability Methods in Deep Neural Networks." NeurIPS. arXiv:1806.10758
- Krishna, S., et al. (2024). "The Disagreement Problem in Explainable Machine Learning." TMLR. arXiv:2202.01602
- Rudin, C. (2019). "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions." Nature Machine Intelligence 1, 206-215. arXiv:1811.10154
Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.