A platform manages infrastructure with code, but engineers make emergency console changes during incidents that are never reflected back. What problems does this drift cause, and how should it be handled without blocking incident response?

The problems drift causes

The code stops describing reality, which breaks the property that made infrastructure-as-code valuable. Every subsequent consequence follows from that:

• The next apply silently reverts the fix. Someone deploys an unrelated change; the tooling sees the console modification as drift and removes it. The incident recurs, now mysteriously, hours or weeks later. This is the most common and most damaging consequence.
• Disaster recovery is compromised. Rebuilding from code produces an environment missing every undocumented fix, so the DR environment differs from production in ways nobody can enumerate.
• Review and audit are bypassed. A change with security or cost implications exists with no record of who made it or why.
• Environments diverge. Staging is built from code, production is code plus accumulated manual changes, and testing stops predicting production behaviour.

Why banning console access is the wrong answer

During a serious incident, the fastest path to restoring service is sometimes a direct change. A policy forbidding it will either be broken under pressure — teaching people that the rules do not survive contact with reality — or obeyed at the cost of extending an outage. Neither is acceptable, and a governance model that only works when nothing is wrong is not a governance model.

The design that works

1. Break-glass access as a designed path, not a violation. Elevated access is available, requires an explicit action to obtain, is time-limited, and is logged loudly. The engineer does not have to choose between the rules and the outage.

2. Automatic drift detection, continuously. A scheduled plan-and-compare across every environment, reporting differences. Drift becomes visible within hours rather than at the next unrelated deployment.

3. Drift creates a ticket automatically, assigned to the team that owns the resource, with the detected difference attached. The obligation is to reconcile — either codify the change or revert it — within a defined window.

4. A reconciliation step in the incident process. The incident is not closed until manual changes are either codified or removed. This is the single highest-value process change, because it attaches the obligation to the moment when the context is still fresh.

5. Make codifying easy. If turning a console change into code takes an hour of fighting the tooling, it will not happen. Import tooling, good module abstractions and fast feedback are what make the correct path the easy one.

6. Detect and refuse silent reversion. The apply pipeline should surface "this apply will remove resources or settings that are not in code" as a prompt rather than performing it silently, so an unrelated deploy cannot quietly undo an incident fix.

The principle

Drift is a feedback signal, not a moral failing. Persistent drift in one area usually indicates that the codified path is too slow, too rigid or missing a capability people need. Treating each instance as a policy violation produces hidden drift; treating it as a signal produces better tooling and, eventually, less drift.