Production is degraded after a release. The team tries to roll back and discovers a migration has already run. What do you do now, and what do you change afterwards?
Show the full answer Hide the answer
Now: stop trying to roll back
Reverting the deployment would run the previous code against a schema it does not know, which is a second and less-understood failure on top of the first. Say this out loud on the call, because the instinct in the room is to keep trying.
Establish two things quickly. What is actually broken — the whole service or one path? A degraded subset may be shed or feature-flagged off, which buys time without touching the deployment. Is there a flag? If the change is behind one, disabling it is the rollback, and it is available even though the deployment is not.
Failing both, it is a forward fix: write, review and deploy a correction under pressure, with the shortest possible change. Resist the temptation to fix the underlying design at the same time.
Afterwards: the design change
The incident is not that a migration ran. It is that a change was shipped whose reversibility nobody had established.
Expand-contract as a rule. Every schema change leaves both the current and the previous application version working: add the new column, write to both, migrate reads in a later release, drop the old shape a release after that. Rollback stays available throughout.
A reversibility statement per change. The deployment record says whether this change can be reverted, and until when. That converts a panicked debate into a lookup at exactly the moment the debate is most expensive.
Rehearse the rollback. A rollback path that has never been exercised is a hypothesis. Include one in a game day.
The measurement that tells you it worked
Mean time to restore, split by whether the incident was resolved by rollback or by forward fix. Healthy delivery resolves most incidents by rollback in minutes. An estate where forward fix is the norm is one where reversibility was never a design constraint, and its restore times will show it.