What an AI Audit Can and Cannot Establish
The three things an audit might mean, why access level determines what conclusions are available, and the gap between certifying a process and certifying an outcome.
"We audited the model" describes at least three different activities with different conclusions, and conflating them is how an audit report gets read as an assurance it never gave.
Three kinds
Governance audit examines whether the organisation's process operated: were assessments completed, approvals recorded, evaluations run, incidents handled. It concludes that decisions were made through the documented process. It says nothing about whether they were good decisions, and it is what a management system certification delivers.
Technical audit examines the artefact: evaluation on held-out data, disaggregated performance, robustness testing, examination of training data and code. It concludes something about the system's properties, bounded by what was tested.
Impact audit examines outcomes in deployment: who was affected, how, and whether the effects differed across groups. It is the only kind that speaks to real-world consequences, and it requires operational data that usually only the deployer holds.
An organisation can pass the first while failing the second, and pass both while failing the third.
Access determines conclusions
Black-box access, meaning query-only, supports behavioural testing, disparity measurement on constructed inputs, and probing for specific failures. It cannot examine training data, cannot distinguish a model property from a filter's effect, and cannot establish absence of a behaviour, only failure to elicit it.
Grey-box access adds logits, embeddings or intermediate activations, enabling calibration analysis and some interpretability work.
White-box access adds weights, training data and code, which is what is needed to examine memorisation, search for backdoors, or verify claims about training data. Almost no external audit gets this.
Deployment access, meaning production logs, decisions and outcomes, is what an impact audit requires and is the rarest of all, since it is the most commercially and legally sensitive.
The practical consequence is that most external audits are black-box, so their conclusions are necessarily about elicited behaviour on tested inputs. A report that reads as broader than its access permitted has overstated itself, and the access level is the first thing to check in any audit document.
When it breaks
Absence of evidence. Failing to elicit a behaviour does not establish it cannot occur, and audit reports frequently phrase a negative result as though it did. "We did not observe X in N tests" is the supportable claim.
Auditors are paid by the audited. The independence problem is the same as in financial audit and has the same distortions, without financial audit's century of regulatory scaffolding around auditor rotation, liability and professional standards.
The scope is chosen by the audited. An audit covers what the organisation asked to be audited. A clean report on a narrow scope is accurate and uninformative about everything else, which is why scope is the second thing to check.
Systems change after the audit. A model retrained monthly has an audit describing a version that no longer exists. Point-in-time assurance for a continuously changing system is worth much less than for a static one, and continuous monitoring evidence is what fills the gap.
12 flashcards for this concept
Click a card to reveal the answer.