Runtime Compliance Drift
also called Plan-Time Blind Spot, Post-Apply Divergence
The growing gap between what infrastructure code declares and what the running estate actually looks like, which a pipeline-based policy check cannot see and therefore reports as compliant.
Every infrastructure pull request passes the policy check. The quarterly scan finds 40 storage buckets without required encryption and 12 with public access. Nothing was merged that would create them.
A policy evaluated against a plan assesses intent at merge time. It says nothing about the state of the account afterwards. Three paths open the gap, and most estates have all three: resources created outside the pipeline with standing credentials, pipeline-created resources mutated by hand during an incident and never reconciled, and resources that predate the policy and are covered by no sweep.
The dangerous property is the direction of the error. A falsely green compliance dashboard earns trust in a way a red one never does, so the drift accumulates behind a signal everyone believes.
Why it matters
Policy-as-code is frequently presented to auditors and boards as the compliance control. If its scope is "changes we chose to route through the pipeline", the assurance it offers is much narrower than the claim being made, and the gap is only discovered by a scan or an incident.
It also misdirects remediation. A quarterly finding of 52 resources produces a clean-up project. The next quarter produces another, because nothing addressed the paths that created them.
Implementation patterns
- Evaluate state continuously, not only plans. Treat plan-time and runtime as two controls with two coverage numbers rather than one control with one green light.
- Admission control at the cloud API, where the platform provides organisation-level guardrails. This is the only placement that prevents rather than reports, because it covers the console and the ad-hoc script as well as the pipeline.
- Continuous reconciliation, so hand-made changes are reverted or flagged in minutes instead of at the next scan.
- Remove standing write credentials in favour of time-limited, logged elevation. Until this is done, every pipeline-based control is advisory.
- Alert on the daily count of runtime policy failures, and on its delta. A quarterly total cannot say when the resources appeared; a daily delta points at the change that caused them.
Industry example
Cloud providers have shipped organisation-level preventive guardrails since around 2018 — policies evaluated by the control plane itself rather than by a pipeline — precisely because pipeline-only enforcement left this gap in every large estate. The pattern predates cloud: network configuration management faced the identical split between the intended configuration file and the device's running configuration, and solved it the same way, with continuous comparison and automatic reversion.
Failure scenarios
- An incident fix at 02:00 that disables a setting, followed by months in which the module is never re-applied, so the code and the account disagree silently.
- A vendor integration granted account-level write access, creating resources that no pipeline ever sees.
- Policy coverage reported as pipeline pass rate, which is 100% by construction and is placed in front of a board as evidence.
- Admission control added without availability engineering, so a policy-service outage becomes an inability to deploy anything — the new failure mode this control introduces.
- Continuous reconciliation enabled estate-wide on day one, reverting a deliberate manual change during an incident and extending the outage.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Plan-time checks only | Cheap; fast feedback to engineers; no runtime dependency | Blind to everything not merged |
| Runtime scanning | Sees actual state; read-only; deployable in days | Detects rather than prevents; findings arrive after the fact |
| Admission control | Prevents creation by any path | A policy service on the creation path, needing production-grade availability |
When not to use it
For a small estate in one account with two engineers, a quarterly scan is a proportionate control and admission control is disproportionate machinery.
The threshold is roughly where the number of people who can create resources exceeds the number who know the policies. Beyond that point drift is produced by ordinary competent work rather than carelessness, and no amount of training reduces it.
Sequence matters more than completeness: scanning first, because it is read-only and cheap; admission control second, once you know from the scan which policies actually matter and how often they would have blocked something. Deploying admission control first blocks a legitimate deployment in its first week and spends the credibility the programme needs.
Interview question
Q: Your infrastructure pipeline reports 100% policy compliance and a scan finds 52 violating resources. Explain the gap, give me the order you would fix it in, and tell me what new risk your fix introduces.
What a strong answer covers: plan-time intent versus runtime state, and the three creation paths that open the gap · why a falsely green dashboard is worse than a red one · runtime scanning before admission control, with the reason being deployability and credibility · admission control as the only preventive placement, covering console and script · continuous reconciliation and its own hazard during incidents · removing standing write credentials as the precondition for everything else · and naming the new risk honestly: a policy service on the creation path is now a production dependency.
Quick check
Quiz: What does a green policy-as-code pipeline actually prove? That no merged change violated the rule — nothing about the current state of the estate, which no plan-time check observes.
Flashcard: Which control prevents rather than reports infrastructure drift? Admission control at the cloud API, because it covers the console and ad-hoc scripts as well as the pipeline — at the cost of a policy service on the creation path.