You roll back a deployment and the system is still broken. What are the likely reasons?

What is being tested

Whether you know that rolling back the application does not roll back the system, and where the irreversible state usually lives.

The likely reasons

1. A database migration the old version cannot read. The most common cause by a wide margin. The new version added a non-nullable column, renamed something, or changed a type, and the previous application version fails against the new schema.

The remedy is expand and contract: every schema change is additive first, so both versions can run against every intermediate state, and the destructive step happens long afterwards as a separate deliberate release.

2. Data written in a new format. Rows, events or cache entries written by the new version that the old version misinterprets or cannot parse. Rolling back the code leaves the data.

3. Messages in flight. Events published in a new schema sitting in a queue, which the rolled-back consumer cannot process. The queue is now a poison pill for the old version.

4. External state changed. A third-party configuration updated, a webhook registered, a subscription created, a file written. None of it reverses with a code rollback.

5. Cache poisoned. The new version populated caches with a new structure; the old version reads it and behaves incorrectly. Frequently overlooked, and the symptom looks like the rollback failing rather than the cache being wrong.

6. It was never the deployment. The correlation was coincidental — a traffic change, a dependency degradation, a scheduled job. Rolling back proved nothing and the actual cause is still running.

The design principle

Every release must be reversible, and reversibility is a property of the whole system, not of the deployment mechanism. Blue-green and canary give you an instant traffic switch, and if the database has moved forward the switch changes nothing.

Concretely:

• Schema changes expand-and-contract, with the contract step weeks later.
• Event and message schemas additive and backward compatible.
• Cache keys versioned, so a new version writes new keys rather than poisoning old ones.
• External side effects idempotent and, where possible, reversible.
• Rollback tested, in a real environment, before you need it.

What to do when it happens

Roll forward with a fix, since backward is closed. Which is why the situation is worth avoiding: your options in an incident have been reduced to the slower one, at the moment you most need the fast one.