Deprecating and Retiring Models

Shipping a model is a well-rehearsed procedure; switching one off is not. A team replaces a ranking model, moves 100% of traffic to the new version and considers the work done. Eight months later a batch job nobody remembers still calls the old endpoint, the GPUs behind it still bill, and the base image it runs on has an unpatched vulnerability. Retirement was never on the plan, so it never happened.

Why models outlive their replacement

A model accumulates consumers the registry does not know about: batch scorers, notebooks, other models that take its output as a feature, a dashboard computing a metric from stored predictions. Each one is a dependency on a specific behaviour. The hidden-consumer problem is old in ML systems; the engineering literature on technical debt singles out undeclared consumers of a model's output as a specific hazard, because they create tight coupling that is invisible until something changes (Sculley et al., 2015, Hidden Technical Debt in Machine Learning Systems, NeurIPS).

So the first retirement task is discovery, and the cheapest instrument is the serving log. If every prediction records the model version and the caller identity, "who still uses version 12" is a query rather than an archaeology project.

The stages of a retirement

A usable lifecycle has more states than "production" and "gone":

1. Superseded. A newer version is the default. The old one still serves. Announce it.
2. Deprecated. Callers receive a warning, in a response header or a log line the caller owns. A published date for removal is attached.
3. Restricted. New callers are refused; existing ones are allow-listed. This stops the consumer count from growing.
4. Retired. Serving is off. The artefact, metadata and evaluation record remain.
5. Purged. Artefact bytes are deleted, if retention policy and regulation allow.

The point of the middle states is that they convert silent breakage into noisy, attributable warnings while there is still time to fix them.

Keeping the ability to explain

Retirement of serving is not retirement of the record. A decision made in March by version 12 may be questioned in October, by an auditor, a customer or a regulator. To reproduce or at least explain it you need the model version, the feature values as they were at the time, and the code and data lineage. Retire compute, keep metadata and logged inputs on the schedule your retention rules require, and decide per model whether the weights themselves must be kept. Where weights are deleted, say so in the record, so that "cannot reproduce" is a documented outcome and not a surprise.

A worked sizing

Suppose version 12 serves 40 queries per second on two GPU replicas, and its replacement takes all interactive traffic. A week later the serving log shows 0.4 qps from three callers: two batch jobs and one notebook. The retirement plan is then concrete: contact three owners, set a removal date a fixed number of weeks out, route the stragglers to the new version with a diff of old and new outputs on their actual inputs so they can see what changes, and scale to a single replica in the interim. The measurable exit criterion is zero calls from non-allow-listed callers over a full business cycle, including month-end, since some jobs only run then.

When it breaks

Month-end consumers. A job that runs quarterly looks absent in a two-week log window. Use a window that covers your slowest cadence.

Models consuming models. If the retired model's output is a feature in another model, switching it off changes that model's inputs. The lineage graph, not the serving log, finds these.

Forced upgrades hide regressions. Moving a straggler to the new version without a comparison swaps one unknown for another. Hand them the diff.

Retention conflicts. Privacy rules may require deleting training data while audit rules require keeping the ability to explain a decision. Decide the order of precedence before the first retirement, not during it.

Related: model lineage across retraining provides the graph used to find downstream dependants, and semantic versioning for models gives the vocabulary for announcing what changes at each step.