Model Drift Monitoring

Models degrade without anything breaking. No error is raised, latency is unchanged, and the outputs are still confident — they are simply becoming less right, because the world moved and the model did not.

The distinctions worth keeping separate. Data drift is a change in the input distribution: a new customer segment, a changed upstream field, a new product category. Concept drift is a change in the relationship between inputs and outcome: the same features now predict a different result, which is what happens after a market shift or a change in fraud tactics. Performance decay is the consequence, and it is measurable only where ground truth eventually arrives.

The monitoring that works accounts for the fact that ground truth is often delayed by weeks — a default, a chargeback, a churn event — so input distribution monitoring is the leading indicator and performance is the lagging confirmation.

For generative systems the equivalents are less crisp and still necessary: distribution of query types, retrieval hit rates, guardrail trigger rates, output length and refusal rates, user-visible signals such as regeneration and thumbs-down, and periodic scored evaluation against a fixed set.

The one specific to third-party models: provider-side change. The model behind an endpoint can be updated without notice, which is an uncontrolled change to your system. A scheduled evaluation run against a fixed set is how you detect it, and it is worth alerting on.