Semantic Versioning for Models and the Breaking-Change Problem

A fraud model moves from version 41 to version 42 on a Tuesday. The code is untouched, the input schema is identical, offline AUC is up. By Thursday the chargeback team is paging someone, because the score distribution shifted: the old model put 3% of traffic above the manual-review threshold and the new one puts 9% there. Nothing crashed, nothing failed validation, and every consumer of the score is now wrong. The version number was an integer that went up by one, and it carried no information about whether anyone downstream needed to react.

What a registry number does and does not say

Registries assign versions mechanically. MLflow's documentation says the first model added under a name becomes version 1 and each later registration under that name increments the number (MLflow, Model Registry). That counter answers "which one is newer", which is useful for ordering and useless for compatibility. Software solved the second question decades ago with a convention: MAJOR.MINOR.PATCH, where a major bump promises a break, a minor bump adds backward-compatible capability, and a patch fixes bugs without changing the contract (Preston-Werner, Semantic Versioning 2.0.0).

The convention transfers to models only once you decide what the contract of a model is. For a library it is the function signatures. For a model it is wider, and most of it is not in the signature.

The surfaces a consumer depends on

Four surfaces matter, and a change to any of them can be breaking:

• Interface. Input feature names, types and order; output shape; label set. A new class in a classifier's output breaks any consumer with a hard-coded switch.
• Semantics. What the output means. A score that moves from "probability of fraud" to "calibrated rank" has the same type and a different meaning.
• Distribution. Where outputs land. Thresholds, alert rules and downstream models are tuned to an output distribution, as the fraud example shows. Calibration changes are the usual culprit.
• Behaviour on slices. Aggregate accuracy can rise while a segment a consumer cares about gets worse.

Interface breaks are cheap to detect: diff the signature. Distribution breaks need a comparison of old and new outputs on the same recent traffic. Slice breaks need an evaluation set stratified by the segments that matter. Semantic breaks need a human to say so.

A workable rule

Treat the interface and semantics as the major version, the distribution and slice behaviour as the minor, and a retrain on fresh data with no measurable shift as the patch. Concretely:

The tolerance is the part that needs a number. "No more than a 1.5 point change in the share of traffic above each published threshold" is checkable; "similar behaviour" is not.

The registry's own integer stays as the immutable artefact identifier. The semantic version rides alongside it as a tag, because tags are key-value pairs attached to a model version (MLflow, Model Registry). Consumers pin to a major line, and the promotion process refuses to move a pointer across a major boundary without the consumers' sign-off.

When it breaks

Tolerances are invented, not derived. A threshold chosen to make the last release pass is not a contract. Derive it from what the most sensitive consumer can absorb.

Model and pipeline versions drift apart. A model whose features come from a pipeline that changed under it has a new behaviour with the same artefact. Version the pair, as in what a model version contains.

Automated retraining erodes the signal. If every nightly retrain is a patch, the history shows 365 patches and one unexplained quarter-long drift. Cumulative shift against a fixed reference release needs its own check, not just the step-to-step one.

Semver gives false comfort for generative systems. Prompt, model and tool changes interact, so a "minor" change can alter behaviour broadly. See versioning compound LLM systems.

For the documentation half of the contract, see model cards and intended use, which is where the declaration of intended semantics lives (Mitchell et al., 2019, Model Cards for Model Reporting, arXiv:1810.03993).