A deployment has caused a production problem. When should the team roll back and when should they fix forward?

The default

Roll back. It restores a known state, it is fast, and it does not require understanding the problem — which matters because mitigation precedes diagnosis, and teams routinely invert that because understanding is more satisfying.

When forward fix is correct

• When rollback is not possible, because a migration has run, external state has changed, or the previous version cannot read data the new one wrote.
• When the fix is trivial and certain — a configuration value, a one-line correction with an obvious cause — and can be deployed faster than a rollback.
• When rolling back would itself cause harm, such as reverting a fix for a worse problem.

"We understand it and the fix is quick" is the reasoning that most often turns a ten-minute incident into a two-hour one, because the fix is deployed, is wrong, and the situation is now less clear than before.

The financial qualification

Mitigation must not create financial ambiguity. Rolling back a service holding in-flight authorisations, or failing over mid-transaction, can convert a service incident into a reconciliation incident that takes weeks to resolve.

The mitigation must account for the state of money in flight, which distinguishes this domain from a general web outage — and which is why connection draining and durable operation records are prerequisites for a safe rollback rather than refinements.

What makes rollback available

• Backwards-compatible changes only, so the previous version can run against the current schema. This is what expand-and-contract migrations buy, and without it rollback is unavailable exactly when it is needed.
• Deployment separated from release, so the fastest reversal is a flag change measured in seconds rather than a deployment measured in minutes.
• Rollback exercised regularly, because one that has never been run under load will fail.
• Artefacts retained, so the previous version is deployable without a rebuild — and a rebuild may not produce the same artefact, which turns a rollback into an unknown.

The decision rule to state in advance

If the cause is not understood within a defined short window, roll back. Agreeing that window before the incident removes the argument from the moment when judgement is worst.