Rollback, Pinning and Reproducible Serving

The single most valuable property of a deployment system is that a bad change can be undone quickly. For application code this is understood and mostly solved. For models it is systematically worse, because a model deployment changes more things than a code deployment and reverting the weights often does not revert the behaviour.

What makes model rollback harder

The bundle problem. If preprocessing, thresholds and prompts were deployed alongside the model, all of them must revert together. A rollback that reverts weights and leaves a new tokeniser in place produces a configuration that has never been tested.

Warm-up cost. Loading a large model, allocating caches and reaching steady-state performance takes minutes. A rollback that must download a hundred gigabytes before serving resumes is not a rollback within any incident's tolerance, which is why keeping the previous version resident, or at least cached on the node, is a design requirement rather than an optimisation.

Stateful side effects. A model that has been writing predictions, populating caches or feeding a feedback loop has left state that reverting the model does not undo. Recommendations logged as impressions, embeddings written to an index, and decisions already actioned all persist.

Learned downstream dependencies. A calibration layer or a downstream model tuned against the new model's output distribution is now mismatched with the reverted one. This is the case that most often turns a rollback into a partial rollback with a second incident.

Mechanisms that help

Immutable, addressable versions. Deploy by digest, never by a moving tag such as latest. The most common cause of an unrepeatable serving configuration is a tag that pointed somewhere different last week.

Traffic-level switching. If the previous version is still loaded and receiving a small share of traffic, reverting is a routing change measured in seconds rather than a deployment. This is the strongest argument for canary deployment beyond risk detection: it leaves the rollback path warm.

Recorded serving configuration. Every response, or a sample of them, annotated with the model version, prompt version and feature versions that produced it. This is what makes it possible to determine after the fact which requests were affected, and without it the blast radius of an incident is a guess.

Rehearsal. A rollback path that has never been executed is a plan. Practising it, ideally as a routine part of deployment, is what turns it into a capability.

When it breaks

Rolling back does not undo the data. Predictions made by the bad model are in the warehouse, in the feature store, and potentially in the next training set. Excluding an incident window from training data requires knowing the window and having the version recorded per prediction.

Forward-fix pressure is often wrong. Under incident conditions the instinct to fix rather than revert is strong and usually costs more, because the fix is written under time pressure and is itself untested. The default should be revert first, diagnose after, and that only works if reverting is genuinely cheap.

Automatic rollback needs a trustworthy trigger. Reverting on a metric threshold sounds good and misfires on noise, on seasonal shifts, and on unrelated upstream incidents. Automatic rollback on hard signals such as error rate or latency is safe; automatic rollback on quality metrics needs enough data to be confident, which usually means it is not fast.

Model rollback and data rollback are different problems. If a bad model was retrained on contaminated data, reverting the model without addressing the data means the next scheduled retrain reintroduces the fault. The retraining trigger has to be part of the incident response.