Sampling Risk
also called Detection Probability, Audit Sampling Power
The probability that a sample-based control test misses a violation that exists in the population - which for small samples and rare violations is most of the time.
An annual audit samples 25 of 10,000 changes. If 5% of changes violate the control, the chance every sampled item is clean is 0.95^25 ≈ 0.28, so the chance of catching at least one violation is about 72%.
Now make the violation rarer, which is what actually happens with serious violations. At a 1% rate the same sample detects something only about 22% of the time. To have a 95% chance of catching a 1% rate you need roughly 300 samples, which no manual audit performs.
Sampling risk is that arithmetic, stated as a property of the test rather than of the population. A clean sample is weak evidence, and it is routinely reported as strong evidence.
Why it matters
Assurance exists disproportionately to find rare, high-impact violations, and those are exactly the ones sampling is worst at detecting. The report says "no exceptions noted", the board reads it as "the control works", and the actual statement is "we looked at 25 of 10,000 and found nothing", which is compatible with a hundred violations.
The second issue is latency. Even when sampling detects, it detects up to a year late. A violation found within a day is a fix; one found eleven months later is an incident report and a restatement.
Implementation patterns
- State the detection probability next to the sample size. One line: "25 of 10,000; detects a 5% rate 72% of the time, a 1% rate 22% of the time."
- Test the full population where the assertion is machine-checkable: every change, every account, every resource. Detection goes to 100% for what the test can express.
- Reserve sampling for judgement, such as whether a change was appropriate or a reviewer was thinking, and draw from a smaller, pre-filtered population.
- Stratify rather than sample uniformly. Test all high-risk changes and sample the routine ones; uniform sampling spends its power on the population you care least about.
- Track time-to-detect as a control metric, not just exception counts.
Industry example
Statistical sampling has been formalised in audit practice for decades, and its mathematics has not changed. What changed is the alternative: when changes, accounts and resources are all API-addressable in production, testing the population costs a scheduled job rather than an auditor's month, which is why continuous controls monitoring displaced sampling for machine-checkable assertions across cloud estates through the 2020s. The mathematics did not make sampling wrong; the availability of full-population testing made it unnecessary for a large class of controls.
Failure scenarios
- "No exceptions noted" read as assurance, when the sample had little power to detect.
- Sampling a period that excludes the risky window — the migration weekend, the year-end freeze — because the sample is drawn uniformly.
- A population defined by the control itself, so violations that escaped the control are outside the sampling frame entirely. This is the failure that makes sampling structurally blind.
- Continuous monitoring reporting 1,800 exceptions on day one, treated as a regression when it is the first honest measurement.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Small sample, annual | Cheap; fits audit calendars | Detects rare violations rarely; up to a year of latency |
| Large stratified sample | Better power where risk is concentrated | Requires a risk model and more auditor time |
| Full-population automated test | Detection near 1 for what it can express; daily latency | Only asserts machine-checkable things; produces a backlog on day one |
When not to use it
Sampling is the right tool when the test requires judgement, when the population is small, or when the control's output is not machine-readable. Reviewing 25 of 40 quarterly access reviews to see whether reviewers engaged is a reasonable use; no automated check answers it. The error is applying sampling to assertions a machine could test in full, and then reporting the result as though the coverage were complete.
Interview question
Q: An auditor reports "we sampled 25 changes and found no exceptions" for a deployment control over a 10,000-change population. The CISO reads this as the control being effective. What do you say, and what would you propose for next year?
What a strong answer covers: computing the detection probability out loud for a plausible violation rate; separating the machine-checkable assertions from the judgement ones; proposing full-population testing for the former and stratified sampling for the latter; naming that the first full run will show a large backlog and that this is a measurement rather than a regression; and avoiding the trap of attacking the auditor, since the sampling was competently performed and the problem is how its result is read.
Quick check
Quiz: A sample of 25 from a large population, with a 1% violation rate: roughly what is the chance of detecting anything? — About 22%, since 0.99^25 ≈ 0.78 is the chance of missing everything.
Flashcard: Which sampling failure cannot be fixed by a larger sample? — A sampling frame defined by the control itself, so items that bypassed the control were never eligible to be drawn.