A deployed model performed well in validation and its business metric has declined over four months. Nothing has been deployed. What do you investigate?

The model did not change; its world did

Three distinct causes, and they need different responses:

Data drift. The input distribution has moved — a new customer segment, a changed upstream field, a seasonal shift the training data did not cover. Detectable by comparing current feature distributions against the training distribution.

Concept drift. The relationship between inputs and outcome has changed. The inputs look the same and what they predict no longer holds — a competitor's pricing change, a regulatory change, a shift in behaviour. Only detectable against actual outcomes.

Pipeline change. Something upstream altered a feature's meaning without altering its schema: a unit, a default, a timezone, a join that started including a category. This masquerades as drift and is the one most often missed, because it is not a model problem at all.

Check the pipeline first

It is the cheapest to rule out and the most likely to be a specific fixable fault. Compare feature values now against a snapshot from launch, field by field. A step change on a specific date points at an upstream deployment, and the date is the evidence you need.

Then check training-serving skew

Were the features at inference computed the same way as at training? Batch SQL for training and application code for serving differ at the edges — null handling, window boundaries, timezone — and those differences never appear in a unit test. This frequently explains a model that was good in validation and mediocre from day one, which a slow decline can disguise.

What should have caught it

Feature distribution monitoring against the training baseline, alerting on material shift.

Outcome monitoring, disaggregated, since aggregate performance hides degradation concentrated in a subgroup.

A retraining trigger defined in advance — a threshold rather than a schedule, so retraining happens when it is needed rather than quarterly regardless.

A champion-challenger arrangement, where a candidate runs alongside on live data, so the decision to replace has evidence.

Monitoring a model is not the same as monitoring the service that serves it, and an inventory entry with no monitoring plan is where this situation begins.