Explaining a Decision to the Person Affected
Why feature attributions are not explanations for a subject, what a counterfactual explanation provides instead, and the gap between technical interpretability and the account a person is owed.
An applicant is declined. They are entitled, in several regimes and under any reasonable standard, to an account of why. What is usually produced is a list of features with importance weights, which answers a question about the model rather than the question the person asked.
Two different questions
"How does the model work?" is answered by feature attributions, partial dependence, surrogate models and the rest of the interpretability toolkit. The audience is a developer, an auditor or a regulator, and the answer is about mechanism.
"Why me, and what would change it?" is answered by something else entirely. The person wants to know what about their situation produced this outcome, whether it was assessed correctly, and what they could do differently. Feature importances do not answer any of those.
Counterfactual explanations
The form that fits the second question is a counterfactual: the smallest change to the input that would flip the decision. "Your application would have been approved with an annual income £3,200 higher, holding everything else constant."
It is intelligible without any understanding of the model, it is directly actionable, and it does not require revealing the model's internals. Wachter and colleagues argued precisely this: counterfactuals provide meaningful explanation to a data subject without opening the black box (Wachter et al., 2018, Counterfactual Explanations Without Opening the Black Box).
Generating good ones requires constraints. The counterfactual must be actionable, changing something the person can change, not their age or their postcode at birth. It must be plausible, lying on the data manifold rather than in a region no real person occupies. It should be sparse, changing few features so the advice is followable. And there are usually many valid counterfactuals, so which is presented is a choice with consequences, since the cheapest change for the model may be the hardest for the person.
When it breaks
Explanations can be gamed. A counterfactual is a specification for changing the outcome, and where the feature is manipulable without changing the underlying reality, publishing it invites exactly that. This is a real tension between transparency and robustness and it is not resolved by picking one.
Post-hoc explanations may not reflect the model. LIME and SHAP approximate a model locally, and different methods disagree on the same prediction. Presenting one as the reason is presenting one approximation's answer, and a subject told two different reasons by two methods has been told the explanation is unreliable.
Consistency matters more than optimality. Two similar applicants given different explanations, because the generator found different counterfactuals, undermines trust more effectively than a slightly worse but stable explanation would. Determinism in the explanation path is worth constraining for.
An explanation is not a justification. Explaining that income drove the decision does not establish that using income that way was appropriate. Regimes requiring meaningful information about the logic involved are asking for something closer to justification, and satisfying them with an attribution list is a common and thin reading.
Systems whose decisions cannot be explained should be questioned rather than annotated. Where no intelligible account of a consequential decision can be given, the appropriate response is sometimes not to deploy, and treating explanation as a box to fill forecloses that conclusion.
14 flashcards for this concept
Click a card to reveal the answer.