Infrastructure is defined as code, but production has drifted from the definitions through manual incident fixes. How should drift be handled, and what is wrong with simply enforcing the code?

Why drift happens and why it is not simply a discipline failure

Manual changes during an incident are correct behaviour. The alternative — going through a pipeline while customers are affected — is worse. Drift is therefore a designed-for condition, not a process violation, and an architecture that treats every manual change as a failure will either be ignored or will slow incident response.

The wrong response

Automatically reverting drift on a schedule. It will undo an emergency fix at 3am and cause a second incident, and it teaches engineers to disable the automation. Enforcement without judgement is worse than detection.

What works

• Detect and report drift continuously, with an owner and an expected resolution time. Visibility is the requirement; automatic correction is not.
• A defined reconciliation path: after an incident, the manual change is either codified (it was right and should persist) or reverted deliberately (it was a stopgap). This should be part of the incident's follow-up, with the same tracking as any other action item.
• Distinguish categories of drift. A manually scaled instance count is benign and often correct. A manually opened security group is a finding. A manually modified IAM policy is an incident. One alert severity for all drift produces noise and then blindness.
• Reduce the need for manual change by making the common emergency actions available through the automated path — a scaling override, a feature flag, a traffic shift — so the pipeline is not the obstacle it is being bypassed for.
• Make the pipeline fast. The main structural driver of drift is that applying a change takes twenty minutes when the incident needs two.

The deeper point

Drift is a signal about the platform, not about the engineers. Persistent drift in one area means the codified path does not serve a real need. The productive response is to ask what the manual changes have in common and to build that capability, rather than to add another control against people doing their jobs during an outage.