Restore-Only Rollback

An upgrade goes badly. The team decides to roll back and discovers there is no rollback: the component does not support downgrade, or a migration has already run, or the previous artefact no longer works against the current data. The only path backwards is a restore.

That is a different operation entirely. A rollback is a deploy of a known artefact, measured in minutes, done weekly. A restore is a procedure written some time ago for slightly different software, never exercised, being read for the first time during an outage.

Why it matters

Rollback availability is the property that makes fast deployment safe. Teams deploy confidently because reversal is cheap; remove reversal and every deployment becomes a one-way door, but the organisation's habits do not change to match.

The cost lands entirely on time-to-restore. Reddit's 14 March 2023 outage ran about 314 minutes, and the published account attributes much of the duration to exactly this: Kubernetes has no downgrade path, so reverting the upgrade meant restoring from a backup using a guide written long ago for different software, never tried in production, and rewritten as they went.

The second-order effect is worse: a team that knows rollback is unavailable will fix forward under pressure, which is the highest-risk activity in software - writing new code, untested, during an incident.

Implementation patterns

• Classify every component by reversal mode before you need to know: deploy-reversible · reversible with a compensating step · restore-only. This is a one-page table and most organisations do not have it.
• Keep the restore-only list short and known, because it determines how much care each change deserves. Changes to restore-only components get more review, smaller batches and longer bake times than anything else.
• Rehearse the restore on a schedule, in production-like conditions, timed. GitLab's January 2017 incident
• where a directory on the primary database was deleted and several backup methods turned out not to have been working, costing roughly six hours of data - is the standing reminder that an unexercised backup is a belief rather than a capability.
• Publish the measured restore time and check it against the recovery objective. If the objective is four hours and the drill takes nine, that is a finding now rather than a discovery later.
• Use expand-and-contract for schema changes so the data remains compatible with the previous version, which converts a restore-only change into a deploy-reversible one.
• Upgrade a copy first. For components with no downgrade, a test cluster carrying the same configuration turns an irreversible production change into a reversible test one.

Industry example

Beyond Reddit's 2023 Kubernetes upgrade, the pattern is the common thread in a large share of long outages: the defect is found quickly and the reversal is not available. Stateful systems are the usual home - database engines with forward-only migrations, message brokers with upgraded on-disk formats, orchestrators without downgrade support - and the duration is set not by diagnosis but by the restore.

Reddit's incident carried two further amplifiers worth noting: the cluster's own metrics died with it, leaving logs as the only signal, and the configuration needed to rebuild had been hand-edited outside version control, so the target state itself had to be reconstructed.

Failure scenarios

• A migration that dropped a column, so the previous application version no longer runs against the data.
• An on-disk format upgraded in place, so the old binary cannot read it.
• A backup that restores but produces an inconsistent state, because the procedure never covered a dependent system.
• Restore time exceeding the recovery objective, discovered during the incident.
• The runbook referring to tooling that no longer exists, written for a previous version of the platform.
• Fix-forward under pressure, producing a second incident on top of the first.
• A restore that requires the configuration nobody has, because the declared state was never declared.

Trade-offs

Preserving reversibility is not free. Expand-and-contract turns one schema change into three or more deployments spread over days. Keeping the previous version compatible constrains what you can change and how fast. Rehearsed restores cost real time and real infrastructure on a schedule, for an event that may not happen this year.

What is bought is the difference between a 20-minute incident and a 5-hour one, on the days it matters - and the option to deploy quickly on all the other days, which is the larger benefit and the one that never appears in the business case.

When not to use it

Where the component genuinely has no state and no migration, reversal is already a deploy and none of this machinery is needed. Stateless services with backward-compatible contracts are the large majority of most estates, and treating them as restore-only wastes the attention that the few genuinely irreversible components need.

There are also changes that are irreversible by nature: data deleted under an erasure obligation, an email sent, a payment settled, an external API version withdrawn. For those, the discipline is not rollback but staging and confirmation - smaller batches, a delay before the irreversible step, and a way to verify before committing.

Interview question

Q: You are about to upgrade the orchestration platform that runs every service in one region. The vendor's documentation says downgrade is not supported. Walk me through how you would plan this.

What a strong answer covers: naming it as restore-only up front and treating that as the central planning constraint · rehearsing the full restore on an identical cluster and publishing the measured time against the recovery objective · verifying that the target state is fully declared, since a restore is useless if the configuration was hand-edited and committed nowhere · upgrading a test cluster with the same configuration and the same version skip, which catches the identifier and API changes that cause most upgrade failures · sequencing so that one region is upgraded while another still serves, so the reversal is traffic shifting rather than a downgrade · checking that observability does not depend on the cluster being upgraded · and an explicit decision point with a named owner for the moment when rolling back stops being possible.

Quick check

Quiz: Why does a component with no downgrade path change how you deploy to it? Because reversal is no longer a deploy but a timed, rehearsed restore - so the change deserves smaller batches, longer bake, a copy upgraded first, and a named point of no return.

Flashcard: What converts a restore-only schema change into a reversible one? — Expand and contract: keep the data readable by the previous version until the new one is proven, so reversal stays a deploy.