Control Testing for AI Systems
How to test whether a stated control actually operates, the difference between design and operating effectiveness, and the AI-specific controls whose testing is unfamiliar to conventional auditors.
Assurance frameworks distinguish two questions about every control. Is it designed such that, if it operated as described, it would address the risk? And did it in fact operate, throughout the period, on every instance it should have? The second is where AI-specific controls fail, and it is where testing is least familiar.
Design versus operating effectiveness
Design effectiveness is assessed by reading the control and reasoning about it. A control requiring human review of high-risk outputs is well designed if the reviewer sees enough context, has authority to override, and is not incentivised against doing so. A control requiring a fairness evaluation before deployment is well designed if it names the metric, the groups and the threshold.
Operating effectiveness is assessed by sampling instances. Take twenty-five deployments in the period and check that each has a completed evaluation, an approval, and a recorded decision. One missing instance is a deviation, and the response is to determine whether it was an exception or evidence the control does not operate.
The gap between the two is where most findings sit. Controls are usually well designed and intermittently operated, because operation depends on people doing something under time pressure.
AI-specific controls and how to test them
Human oversight. Test by sampling decisions and checking that a human reviewed them, and separately by measuring the override rate. An override rate of zero is evidence the control is nominal, since a reviewer who never disagrees is not exercising judgement.
Evaluation before deployment. Test by taking deployed model versions from the registry and checking each has evaluation results predating deployment, on a recorded dataset version, meeting the stated threshold.
Monitoring and alerting. Test by checking that alerts fired for known incidents, and, more informatively, by injecting a synthetic condition and confirming detection. A monitoring control that has never been tested end to end is a configuration, not a control.
Data quality gates. Test by finding instances where a gate should have blocked and confirming it did, which requires the gate to log its decisions rather than only its failures.
Incident response. Test by walking a recent incident from detection to resolution and checking the timeline against the stated process, including whether reportability was assessed.
When it breaks
Sampling assumes homogeneity. Twenty-five samples from a population where one subgroup behaves differently will likely miss it. Stratified sampling across model types, risk tiers and teams is what makes a sample informative about a heterogeneous estate.
Controls that only log failures cannot be tested. If a gate records nothing when it passes, there is no evidence it evaluated anything. Logging every decision, not just the negative ones, is a precondition for the control being testable at all.
Automated controls are tested once and assumed forever. A pipeline check that worked at implementation can be silently disabled by a configuration change or a refactor. Periodic re-testing, ideally by injecting a condition that should fail, is what catches that.
Compensating controls are claimed and not tested. When a control is weak, organisations point to another that supposedly covers the gap. That claim needs testing on its own terms, and it usually turns out that the compensating control was not designed for the risk it is being credited with covering.
12 flashcards for this concept
Click a card to reveal the answer.