During an incident an engineer edits a live Kubernetes resource and the change is reverted a minute later by the GitOps controller. The incident continues. What is your position?

The controller behaved correctly

Continuous reconciliation is the property that makes drift impossible and makes the repository an accurate description of the estate. It reverted an out-of-band change because that is what it exists to do.

The failure is that nobody had designed the emergency path, so it was discovered at the worst possible moment.

What should exist

A documented pause mechanism. Every GitOps controller supports suspending reconciliation for a specific resource or namespace. The command, its scope and its side effects belong in the incident runbook, tested in a game day rather than read for the first time under pressure.

A bound on the pause. Suspension that outlives the incident is how drift returns permanently. Either an automatic expiry or, at minimum, an alert while any reconciliation is suspended, so a forgotten suspension is visible.

A path back. After the incident, the change is either committed to the repository — making it the declared state — or discarded and reconciliation resumed. Deciding which is part of closing the incident, not a follow-up task.

The better answer where it applies

Most emergency changes fall into a small number of shapes: scale something, disable a feature, shed traffic, raise a limit. Each can be a declared, committed change that goes through the normal path — which for a well-tuned GitOps setup is a commit and a reconciliation, measured in a minute or two.

If that path is too slow to use during an incident, that is the finding worth acting on. Fixing it improves every change rather than only emergencies, and it removes most of the demand for a bypass.

What not to conclude

Not that GitOps is unsuitable for production. The alternative — a push model where anyone with credentials can change the cluster and nothing detects it — has the opposite failure: the emergency change works, stays forever, and nobody knows the running state differs from the repository until a rebuild fails.