During an incident an engineer makes a manual change in the cloud console. It resolves the issue. What should happen next and what should have prevented it?

What is being tested

Whether you treat drift as a defect with a defined response, rather than as a rule violation to scold someone about.

First: the change was correct

Making the manual change during an incident was the right call. Restoring service beats process purity, and any culture that punishes it will produce engineers who hesitate during outages, which is far more expensive. The problem is not the change; it is the change remaining unreconciled.

What should happen next, in order

1. Record it immediately — in the incident channel, with what was changed and why. The most common failure is that the change is forgotten within a day and rediscovered months later when an apply reverts it and causes a second outage.

2. Reconcile within a defined window. Either update the code to match the change, or revert the change and implement it properly. Both are acceptable; leaving the divergence is not. A 48-hour reconciliation commitment as part of incident follow-up works better than a general aspiration.

3. Verify the reconciliation with a plan showing no unexpected differences.

4. Ask why the code path was too slow. If the emergency change had to be manual because the pipeline takes 40 minutes, the pipeline is the finding. An emergency path through the code — expedited approval, fast-track apply — is what stops the console from being the only option under pressure.

What should have prevented it

• Automated drift detection, running continuously, alerting when reality diverges from code. This is what turns "someone will remember" into a signal.
• Read-only console access in production for routine work, with break-glass elevation that is time-limited, logged and automatically notifies the team. This does not prevent the emergency change; it makes it visible and temporary by construction.
• A fast emergency apply path, so the correct route is also the quick one.
• Modules with narrow interfaces, so the common changes are parameter edits rather than intimidating configuration rewrites.

The related practices worth naming

Remote state with locking, because two engineers applying concurrently during an incident is how infrastructure gets destroyed. Separate state per environment and per blast radius, so an apply does not risk everything. Plan output reviewed in the pull request, so a reviewer sees that a change replaces a database rather than modifying it.

The framing

Infrastructure as code is only true if the code describes reality. One unreconciled manual change does not break that; a culture of unreconciled manual changes converts the repository into documentation of intentions — which is exactly the failure mode that infrastructure as code existed to fix.