Drift Statistics and What They Miss
PSI, KL divergence, KS and MMD compared on what they detect and where they fail, why per-feature tests miss joint shifts, and the multiple-comparison problem that makes wide monitoring noisy.
Detecting distribution shift means reducing two samples to a number and comparing it to a threshold. Every choice of number emphasises something different, and each has a regime where it is blind.
The common measures
Population Stability Index bins both distributions and sums \((p_i - q_i)\ln(p_i / q_i)\) over bins. It is the industry standard in credit risk, its conventional thresholds are 0.1 for moderate and 0.25 for significant shift, and those thresholds are heuristics from practice rather than results from theory. It is symmetric, interpretable, and entirely dependent on the binning: too few bins hide shifts within a bin, too many make it unstable on small samples.
KL divergence is asymmetric and unbounded, infinite when the new distribution has mass where the reference has none, which is exactly the case of a genuinely novel input. That sensitivity is either the point or a nuisance depending on the application. Jensen-Shannon divergence symmetrises and bounds it.
Kolmogorov-Smirnov compares empirical cumulative distributions and needs no binning, which removes PSI's main arbitrary choice. It is most sensitive near the centre of the distribution and comparatively blind in the tails, where model failures often live. It applies to one continuous variable at a time.
Maximum Mean Discrepancy compares distributions in a kernel feature space and works multivariately, so it detects changes in the joint distribution that no per-feature test can see. It costs more, needs a kernel and bandwidth choice, and gives a number that is harder to interpret than PSI's.
For categorical features, chi-squared tests and the appearance of previously unseen categories are the relevant checks, and the second is often more actionable than any divergence.
What per-feature monitoring misses
Every marginal can be unchanged while the joint distribution moves substantially. If height and weight are individually distributed as before but their correlation has inverted, no per-feature test fires and the model, which learned from their joint structure, is now operating on inputs it never saw.
Two practical detectors handle this. Train a classifier to distinguish reference from current samples: if it can do better than chance, the distributions differ, and its accuracy is an interpretable measure of how much. Its feature importances then say which variables carry the difference. Alternatively monitor the distribution of the model's own predictions, which is a learned one-dimensional projection of the joint distribution and moves when anything the model actually uses moves.
When it breaks
Sample size drives significance, not importance. With a million samples, a statistical test rejects on differences too small to matter. Effect sizes and fixed-size samples are more useful than p-values for monitoring, and this is the single most common misuse of KS tests in production.
Monitoring 200 features means 200 tests. At a 1 percent false positive rate that is two alerts per run from noise alone. Correction for multiplicity, or ranking features by drift magnitude and inspecting the top few rather than alerting on each, is required for the system to remain readable.
Thresholds are borrowed rather than derived. PSI's 0.1 and 0.25 come from credit scorecard practice, and there is no reason they transfer to a different feature type or a different model. Calibrating against your own historical drift, and against drift levels that did and did not degrade performance, produces thresholds that mean something.
Reference windows go stale. A fixed reference from training becomes less representative every month, so drift scores climb steadily and eventually everything is drifting. Refreshing the reference resets that and destroys the ability to see cumulative change, so both a fixed and a rolling reference are needed.
14 flashcards for this concept
Click a card to reveal the answer.