Local Explanations: LIME, SHAP and Their Limits

Slack and colleagues built a recidivism classifier on the COMPAS data that decided using race alone, then wrapped it so that LIME and SHAP would report something else. On held-out defendants, the top-ranked feature moved from race to a synthetic, uncorrelated feature in 100 percent of LIME explanations and 84 percent of SHAP explanations, while the wrapped model's decisions on real inputs were still driven by race (Slack et al., 2020, Fooling LIME and SHAP, AIES, arXiv:1911.02508). The attack works because of how both methods generate their evidence, and understanding that mechanism is the fastest route to knowing what a local explanation can support.

LIME: a weighted surrogate around one point

LIME explains a prediction \(f(x)\) by fitting a simple model \(g\) from an interpretable class \(G\), usually sparse linear, to the black box in a neighbourhood of \(x\) (Ribeiro, Singh & Guestrin, 2016, "Why Should I Trust You?", KDD, arXiv:1602.04938):

where \(\mathcal{L}\) is a locality-weighted squared error over perturbed samples \(z\), \(\pi_x(z)\) is a proximity kernel, and \(\Omega\) penalises complexity. In the reference tabular implementation the kernel is \(\exp\!\big(-D(x,z)^2 / 2\sigma^2\big)\) with default width \(\sigma = 0.75\sqrt{p}\) for \(p\) features, and 5,000 samples are drawn by default. With 16 features the width is 3.0.

Every design choice there is a free parameter. The kernel width decides what "local" means; the sampling distribution decides which synthetic points the surrogate is fit on; the sparsity level decides how many features appear. Change any of them and the explanation changes, with nothing in the output to say which setting was right.

SHAP: Shapley values from sampled coalitions

SHAP treats features as players in a cooperative game whose payout is the prediction. The Shapley value, introduced by Shapley in 1953, is the unique allocation satisfying four axioms: efficiency (attributions sum to the prediction minus a baseline), symmetry (interchangeable features get equal credit), dummy (a feature that never changes the payout gets zero), and additivity (attributions for a sum of games add). Lundberg and Lee showed that within additive feature attributions, only Shapley values satisfy their local accuracy, missingness and consistency properties (Lundberg & Lee, 2017, A Unified Approach to Interpreting Model Predictions, NeurIPS, arXiv:1705.07874).

Exact values need the model's output on every subset of features, \(2^M\) coalitions: over a million at \(M = 20\). KernelSHAP samples coalitions \(z'\) and solves a weighted linear regression with the Shapley kernel

which puts most weight on very small and very large coalitions, where a single feature's marginal effect is most informative. Missing features are filled with draws from a background dataset. That makes KernelSHAP an estimate with sampling variance, and makes the answer depend on the background: the same prediction explained against all applicants or against approved applicants gets different attributions. Exact, fast computation exists only for particular model classes, as in TreeSHAP for tree ensembles.

Why they disagree and can be fooled

Both methods query the model on synthetic points: LIME's perturbations and SHAP's background-filled coalitions. Many of those points lie off the data manifold, in combinations no real input occupies. Slack's scaffold exploits exactly this. It trains a detector for perturbed inputs and routes them to an innocuous model \(\psi\), so the deployed classifier is \(e(x) = f(x)\) on realistic inputs and \(\psi(x)\) otherwise. SHAP resisted somewhat better because efficiency forces attributions to add up across features.

Without an adversary, the methods are still fragile. Alvarez-Melis and Jaakkola found that popular explainers violate a basic robustness expectation, that similar inputs receive similar explanations (Alvarez-Melis & Jaakkola, 2018, On the Robustness of Interpretability Methods, arXiv:1806.08049). Across four datasets, six models and six explanation methods, Krishna and colleagues found that explainers frequently disagree on the same prediction, and that practitioners resolve the conflict with ad hoc heuristics (Krishna et al., The Disagreement Problem in Explainable Machine Learning, TMLR, arXiv:2202.01602). Bilodeau and colleagues went further and proved that complete, linear attribution methods, a class including SHAP and Integrated Gradients, can provably fail to beat random guessing on end tasks such as detecting spurious features for sufficiently expressive models (Bilodeau et al., 2024, Impossibility Theorems for Feature Attribution, PNAS 121(2), arXiv:2212.11870).

When it breaks

The field disagrees about whether to explain black boxes at all. Rudin argues that for high-stakes decisions, post-hoc explanations of opaque models perpetuate bad practice and inherently interpretable models should be used instead, often at little accuracy cost (Rudin, 2019, Nature Machine Intelligence 1:206-215, arXiv:1811.10154). The usual reply is that interpretable models rarely match deep networks on text and images, where post-hoc explanation is the only option available. Where the accuracy gap is small, Rudin's position is hard to answer.

Attribution is not causation. A high SHAP value says the model's output moved when the feature was replaced by background values. It does not say changing that feature in the world would change the outcome, which is the question a subject cares about (explaining a decision to the person affected covers that gap).

An audit that relies on explanations can be gamed. If a regulator checks for use of a protected attribute by inspecting LIME or SHAP output, Slack's result shows a motivated provider can pass while discriminating. Explanations are evidence to triangulate with direct behavioural tests, not a certificate.

Stability must be measured, not assumed. Re-run with different seeds, sample counts and backgrounds before reporting any attribution; if the top features reorder, the explanation is noise.