A platform manages infrastructure declaratively and production has diverged from the declarations. Is that a failure of discipline?

Not a failure of discipline

Manual changes during an incident are correct behaviour. Going through a twenty-minute pipeline while customers are affected is worse. Drift is a designed-for condition, and an architecture treating every manual change as a violation will either be ignored or will slow incident response.

The wrong response

Automatically reverting drift on a schedule. It will undo an emergency fix at 3am and cause a second incident, and it teaches engineers to disable the automation — which removes the detection along with the enforcement. Enforcement without judgement is worse than detection.

What works

• Detect and report continuously, with an owner and an expected resolution time. Visibility is the requirement; automatic correction is not.
• A defined reconciliation path after every incident: the manual change is either codified because it was right, or reverted deliberately because it was a stopgap — tracked as an incident action item.
• Categorise drift by risk. A manually adjusted instance count is benign and often correct; an opened security group is a finding; a modified identity policy is an incident. One severity for all drift produces noise and then blindness.
• Reduce the need for manual change by exposing the common emergency actions through the automated path — a scaling override, a feature flag, a traffic shift.
• Make the pipeline fast, because the main structural driver of drift is a twenty-minute apply during an incident that needs two.

The deeper reading

Drift is a signal about the platform, not about the engineers. Persistent drift in one area means the codified path does not serve a real need, and the productive response is to ask what the manual changes have in common and build that capability.

The honest limit

Drift is acceptable for hours and corrosive over months, because every guarantee built on the declaration — compliance evidence, disaster recovery, reproducibility — weakens as the declaration becomes fiction.

Which is why the resolution clock matters more than the detection.