Half your production changes are made through the console during incidents. The IaC repository no longer matches reality. What do you do?

Why this matters beyond tidiness

The code has stopped being a description of reality, which removes the properties it was adopted for: the ability to recreate an environment, to diff two environments, and to review a change before it happens. The next apply either reverts an incident fix — reintroducing the incident — or reports a difference nobody understands, so applies stop being trusted and the drift accelerates.

Detect first, so the scale is known

Run plan without apply on a schedule across every environment and report the differences. Most tools support this and several providers offer native drift detection. This turns an invisible accumulation into a number, which is what makes the case for the next step.

Then prevent, because detection alone does not stop it

Remove console write access in production. Read-only for investigation, which is what people actually need during an incident far more often than write.

Provide a break-glass path that is audited, time-limited and alerting — because a policy with no emergency route gets circumvented entirely the first time it blocks something urgent, and then you have neither the control nor the visibility.

Require reconciliation into code within a defined window, with a follow-up ticket created automatically when break-glass is used. Relying on someone remembering afterwards is how the current situation arose.

Reconcile what already exists

For the drift already present: for each difference, decide whether the console change was correct. If it was, import it into code. If not, let the next apply revert it, deliberately and in a change window rather than by surprise.

Prioritise by blast radius rather than doing it alphabetically — security groups, IAM and networking first.

The structural change worth making at the same time

Split state by blast radius. One enormous state file means every change plans against everything, applies are slow and frightening, and one corruption affects everything. Separate state per environment and per major component makes routine changes small and safe, which is itself a large part of why people reach for the console.

What a strong answer adds

Recognising that heavy console use during incidents is usually a symptom of slow or untrusted pipelines, not of indiscipline. If applying a fix through code takes twenty minutes and the console takes two, people will use the console during an outage and they are right to. Making the pipeline fast is often the intervention that actually changes behaviour.