A change causes a global incident. When should you roll back and when should you fix forward, and what property makes the choice easy?

The default

Roll back. It is fast, predictable, and restores a state that was known to work. Fixing forward under incident pressure means writing and shipping code with a degraded team in a hurry — which is how second incidents happen.

When forward-fix is correct

• When rollback is not possible — the change has taken writes the previous version cannot read, or an irreversible migration has run.
• When the previous version has a worse defect — a security issue, or a bug the change was fixing.
• When the fix is trivially small and verified, and rollback would be materially more disruptive.

Each of these should be a deliberate judgement, not a default, and the decision should be made by a named person against a pre-agreed trigger — because rollback decisions made under pressure without a trigger are made late, and lateness is what turns a recoverable incident into a long one.

The property that makes the choice easy

Rollback must be faster than rollout, and must not depend on whatever broke.

That means: a locally cached last-known-good version, an automatic revert on failure to receive a heartbeat, and a rollback path that does not require the control plane a bad configuration may have disabled.

If a bad change breaks connectivity to the system that would undo it, the system is not rollback-capable — and that is the single most important property to verify for anything that propagates globally.

What makes rollback impossible, and how to avoid it

• Schema changes that are not backward compatible. Expand-and-contract makes every intermediate state rollback-safe, which is the main reason to use it.
• Data written in a new format the old version cannot read. Dual-write during transition.
• Irreversible external effects already dispatched — messages sent, payments taken. These need compensation rather than rollback, which is a different and slower process.

The structural mitigation

Blast-radius staging. A change reaching a canary location, then a region, then the fleet — gated on served-traffic health at each step — means a bad change affects a bounded population before it affects everyone, and the rollback is correspondingly smaller.

The danger window is proportional to how long undoing takes, which is why rollback speed is a design requirement rather than an operational nicety.