Population Completeness

An assessor asks for every production deployment in the twelve-month period. A team exports 9,412 rows from the deployment system and feels well prepared. The assessor's first question is not about any row; it is how you know there were 9,412 and not 9,600.

The sample is only as good as the list it came from. If deployments can reach production by a path the export does not cover - a manual console action, a break-glass pipeline, a second CD system inherited with an acquisition, an infrastructure change applied directly - then the sample is drawn from the well-behaved subset and tests the control exactly where it was already working. This is the most common reason a clean sample result coexists with a later finding.

Completeness is a property of the evidence, not of the control, which is why teams that have automated their controls beautifully still fail on it.

Why it matters

Sampling is the economic basis of assurance: 25 to 40 instances stand in for thousands. That substitution is valid only if every instance had a chance of being selected. An incomplete population breaks the inference silently, and the direction of the error always flatters the organisation, because the paths that are missing from the list are the ones with the weakest controls.

It also changes what you build. A control designed to produce its own complete population is a different design from one whose evidence is queried out of a general-purpose log afterwards.

Implementation patterns

• Three-source reconciliation. The control's own records, an independent record of the effect, and an inventory of the actors. For deployments: CD run records, the artefact digests actually running according to the orchestrator, and the list of identities with deploy permission. Each pair that disagrees is a finding before any sampling happens.
• Close the paths or count them. Every route to production either flows through the recorded path or emits its own record. Break-glass stays open and is counted: break-glass as a share of all changes is the single number that tells you how much of the population the control covers.
• A completeness statement shipped with the evidence: the query, the systems it covers, the paths it does not, and the reconciliation result for the period. It takes a page and it is what turns an export into evidence.
• Monotonic sequence numbers per control instance, so a gap in the sequence is visible without a second system to compare against.
• Thresholds and actions. A reconciliation variance above 1% for two consecutive weeks is a control deficiency to raise yourself, not something to discover during fieldwork.

Industry example

Published postmortem write-ups keep showing the same shape: a change reached production through a path the organisation's own change data did not describe - an emergency pipeline, a console action, a configuration push classed as data rather than as a release. The incident is attributed to the change; the assurance lesson is that the population used all year to demonstrate the control did not contain that class of change at all.

Failure scenarios

• The second pipeline. An acquired team's CD system was never added to the export, so 11% of deployments are outside the population and nobody noticed for two audit cycles.
• The silently narrowed query. The evidence query filters on a project tag; teams that stopped applying the tag vanished from the population while the row count kept rising.
• Retention shorter than the period. The population is complete for the last 90 days of a 365-day window, and the first nine months cannot be reconstructed.
• Completeness confused with accuracy. Every instance is present and the records misdescribe what happened, which reconciliation against an independent effect would have caught and a row count never will.

Trade-offs

When not to use it

Where the control's own evidence is an absence rather than a list, completeness is the wrong test. Showing that nobody used a privileged role needs continuity of logging and tamper resistance; counting a population of zero proves nothing. And proportionality holds here as everywhere: a three-source reconciliation for a low-consequence control costs more than the assurance is worth, so reserve it for the controls a regulator or a customer will actually sample.

Interview question

Q: An assessor asks for every production change in the last twelve months. You hand over an export. How do you demonstrate it is complete, and what would you expect to find when you first try?

What a strong answer covers: completeness precedes sampling, and an incomplete population makes a clean sample meaningless · reconciliation against an independent record of effect plus the identity inventory · break-glass counted rather than excluded · the completeness statement as a deliverable · the honest expectation that the first reconciliation finds a second deployment path, an untagged team and a retention window shorter than the period.

Quick check

Quiz: Why can a control pass a 40-instance sample and still be ineffective? Because the sample was drawn from an incomplete population, and the paths missing from it are the ones where the control was never operating.

Flashcard: Which number tells you how much of a change population your gate actually covers? - Break-glass or out-of-band changes as a share of all changes; a gate covering 90% of changes provides assurance over 90% of them and none over the rest.