The Three Drifts and How They Differ
Covariate shift, label shift and concept drift decomposed precisely, why only one of them necessarily degrades a model, and which of them your monitoring can actually see.
"The model is drifting" describes three different things with three different consequences and three different detection requirements. Collapsing them into one word is why teams monitor the wrong quantity, alert on changes that do not matter, and miss the change that does.
Write the joint distribution two ways, \(P(X, Y) = P(Y \mid X)P(X) = P(X \mid Y)P(Y)\), and the taxonomy falls out of which factor moved.
Covariate shift
\(P(X)\) changes; \(P(Y \mid X)\) does not. The inputs look different and the relationship between inputs and outcome is intact. A new customer segment arrives, a marketing campaign changes traffic mix, a device type becomes popular.
This is the drift everyone monitors, because it needs no labels: compare the feature distribution now against a reference window. It is also the drift least likely to hurt, since a model that was correct on the old distribution is still correct wherever the two overlap. Damage occurs only where the new inputs fall outside the region the model was fitted on, so the useful measurement is not "did the distribution move" but "how much mass is now in regions with little training support".
Label shift
\(P(Y)\) changes; \(P(X \mid Y)\) does not. The base rate moves. Fraud prevalence rises, disease incidence changes seasonally, a class becomes more common.
This breaks calibration directly, since a classifier's outputs encode the prior it was trained under. Ranking usually survives, so a model used for ordering may be unaffected while the same model used for thresholded decisions is badly wrong. The correction is cheap when the new prior can be estimated: reweight the predicted probabilities by the ratio of new to old class priors, without retraining anything.
Concept drift
\(P(Y \mid X)\) changes. The same inputs now imply a different outcome. User preferences change, an adversary adapts, a business rule changes what counts as a positive.
This is the only one that necessarily degrades the model, and it is the one input monitoring cannot detect, because \(P(X)\) may not move at all. Detecting it requires labels, or a proxy for them, and where labels arrive with a long delay the drift is invisible for exactly as long as that delay.
When it breaks
Input drift detection has a bad signal-to-noise ratio. Most detected covariate shift is harmless, so thresholds get widened until the detector stops firing, at which point it also stops detecting the rare harmful case. Monitoring model outputs is usually more informative per alert, since a shifted prediction distribution reflects whatever combination of causes actually reached the model.
Label delay decides what is monitorable. With labels in minutes, monitor performance directly and treat drift metrics as diagnostics. With labels in months, drift proxies are all you have, and their limitations should be stated rather than assumed away.
Real shifts are mixtures. A new market segment brings both different features and a different feature-outcome relationship. The taxonomy is a diagnostic aid for reasoning about causes, not a partition that production data respects.
The reference window is a choice with consequences. Comparing against training data detects total accumulated drift and fires forever after a permanent legitimate change. Comparing against a rolling recent window detects sudden change and adapts to gradual drift so completely that it never reports it. Both are needed, and reporting only one is how slow degradation stays invisible.
12 flashcards for this concept
Click a card to reveal the answer.