Drift Reconciliation

Manual changes during an incident are correct behaviour. Going through a twenty-minute pipeline while customers are affected is worse than making the change directly. Drift is therefore a designed-for condition, not a process failure, and an architecture that treats every manual change as a violation will either be ignored or will slow incident response.

Why it matters

The naive response — automatically reverting drift on a schedule — will undo an emergency fix at 3am and cause a second incident. It also teaches engineers to disable the automation, which removes the detection along with the enforcement. Enforcement without judgement is worse than detection.

Implementation patterns

• Detect and report continuously, with an owner and an expected resolution time. Visibility is the requirement; automatic correction is not.
• A defined reconciliation path after every incident: the manual change is either codified, because it was right and should persist, or reverted deliberately, because it was a stopgap. Tracked as an incident action item with the same follow-up as any other.
• Categorise drift by risk. A manually adjusted instance count is benign and often correct. An opened security group is a finding. A modified identity policy is an incident. One severity for all drift produces noise and then blindness.
• Reduce the need for manual change by exposing the common emergency actions through the automated path — a scaling override, a feature flag, a traffic shift — so the pipeline is not the obstacle it is being bypassed for.
• Make the pipeline fast, because the main structural driver of drift is that applying a change takes twenty minutes when the incident needs two.

Industry example

Developer-infrastructure platforms such as Render and Railway operate large fleets where both the platform's own infrastructure and customers' workloads are declaratively managed. The pattern that separates mature operations from fragile ones is not the absence of drift — it is whether drift has an owner and a resolution clock.

The same is true in any organisation running infrastructure as code at scale: drift accumulates in proportion to incident frequency and pipeline slowness, and those are the two variables worth acting on.

Failure scenarios

• Automatic revert on a schedule, undoing emergency fixes.
• Drift detected and never resolved, so the declared state becomes fiction and nobody trusts it.
• Uniform alerting, producing noise that trains people to ignore genuine security drift.
• A slow pipeline, guaranteeing bypass under pressure.
• Drift treated as a discipline problem, generating process and blame rather than the capability that would remove the need.

Trade-offs

Tolerating drift means the declared state is not always the actual state, which weakens every guarantee built on the declaration — compliance evidence, disaster recovery, reproducibility. That is a real cost, and it is why the resolution clock matters: drift is acceptable for hours and corrosive over months.

The opposite extreme — no manual change permitted, ever — is defensible only in environments where the pipeline is fast enough to be used during an incident, which is a capability rather than a policy.

Interview question

"Your drift detector shows forty differences in production. How do you triage them, which do you fix this week, and what would you change so there are fewer next quarter?"