Drift Is Not Decay: What Production Shift Detectors Actually Tell You

On Camelyon17, a tumour-classification benchmark built from slides scanned at five hospitals, a standard model scored 93.2% on in-distribution validation and 70.3% on the held-out hospital (Koh et al., 2021, WILDS, ICML, arXiv:2012.07421). Twenty-three points, lost to a change of scanner and staining protocol. Nothing in the pixel statistics announces itself as catastrophic; a pathologist would not notice. The model does.

Now the awkward part. Run any competent drift detector on that same data and it fires, correctly. Run the same detector on a hundred other hospital transfers where accuracy barely moves, and it fires there too, just as correctly. The detector is answering the question it was asked, which is whether the input distribution changed. Nobody deploying it wanted the answer to that question.

Why this matters: Almost every production ML stack pages on input drift and computes true performance in a quarterly review. That ordering is backwards. Input drift is a diagnostic that explains what changed; it is a poor trigger for that something is wrong, and the arithmetic of daily testing guarantees it will be ignored within weeks. The signal worth building the monitor around is risk, and there is now a decade of method for estimating it without labels and testing it without invalidating yourself by looking.

TL;DR

• Forty features, a KS test each, \(\alpha = 0.05\), run daily: a perfectly healthy system alarms with probability \(1 - 0.95^{40} \approx 87\%\) every morning. Bonferroni within the day still leaves roughly a 99% chance of at least one alarm per quarter.
• Statistical significance on inputs is not effect size on loss. Any two-sample test rejects on an arbitrarily small real difference once \(n\) is large, and production input distributions always contain small real differences.
• Input monitoring is also blind in the other direction. When \(P(y \mid x)\) moves while \(P(x)\) holds, which is what an adapting adversary produces, there is nothing in the inputs to detect.
• Detection got solved, and it was not the bottleneck. Feeding a pre-trained classifier's softmax outputs into a two-sample test detected shift from as few as 10 target samples (Rabanser et al., 2019).
• Estimating current accuracy without labels works better than practitioners expect. A single confidence threshold fit on source data estimated target performance 2 to 4 times more accurately than prior methods across WILDS, ImageNet, BREEDS, CIFAR and MNIST (Garg et al., 2022).
• Continuous monitoring needs time-uniform statistics, not p-values. Confidence sequences and conformal test martingales stay valid under unlimited peeking; a daily KS test does not.
• The PSI bands everyone uses, 0.10 and 0.25, were adopted without reference to Type I or Type II error rates and depend on bucket count and sample size.

At a Glance

flowchart LR
    W["World changes"]:::blue
    I["Input drift"]:::slate
    O["Score drift"]:::slate
    R["Realised risk"]:::rose
    E["Estimated risk<br/>no labels"]:::purple
    S["Sequential test"]:::purple
    A["Alarm with<br/>an action"]:::teal
    D["Diagnosis:<br/>what changed"]:::amber
    W --> I --> O --> R
    O --> E --> S --> A
    R --> S
    A --> D
    I -.->|"explains, does not page"| D
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

The causal chain runs left to right, and the further left you monitor, the more of what you see is noise. Input drift belongs on the diagnosis path, not the paging path.

[IMAGE: Two side-by-side alert timelines over 90 days. Top, labelled "input drift monitor": a dense forest of ~80 alert markers with three shaded bands marking real incidents, indistinguishable from the rest. Bottom, labelled "sequential risk monitor": four alert markers, three of which fall inside the shaded bands. Caption: "Same system, same incidents, two monitoring designs."]

Before the Alarm Had a Meaning

Distribution shift entered machine learning as a theory problem, not an operations problem. Shimodaira's importance-weighted log-likelihood gave the canonical treatment of covariate shift and the canonical fix: reweight training examples by the density ratio (Shimodaira, 2000, JSPI 90(2), 227-244). That framing assumed you knew a shift had occurred and had samples from both sides. Detection was somebody else's problem.

The data-stream community took it on, and it took it on from the direction that turns out to be right. Gama and colleagues' drift detection method watches the online error rate of a learner and signals when it degrades past a control limit, which is a risk monitor in everything but name (Gama, Medas, Castillo & Rodrigues, 2004, SBIA). ADWIN followed with an adaptive window that grows while the stream looks stationary and shrinks when a statistical test finds a split point, with explicit false-positive and false-negative guarantees (Bifet & Gavaldà, 2007, SIAM SDM). Both assume labels arrive with the stream. Sequential change detection itself is older still, going back to Page's CUSUM (Page, 1954, Biometrika).

Then supervised learning went to production at scale, labels stopped arriving promptly, and the industry reached for what it could compute: two-sample tests on features. Kernel methods gave the general-purpose tool, with the maximum mean discrepancy providing a distribution-free test for whether two samples share a distribution (Gretton et al., 2012, JMLR 13, 723-773). Credit risk contributed the population stability index, a binned symmetrised KL divergence with two memorable thresholds. Google shipped data validation into TFX and hundreds of product teams used it to check petabytes of production data daily (Breck et al., 2019, Data Validation for Machine Learning, SysML).

What none of that answered is whether a detected change is worth waking anyone for.

[IMAGE: A two-track horizontal diagram. Upper track, labelled "data streams research", runs DDM to ADWIN to conformal martingales, annotated "monitors the error rate". Lower track, labelled "production ML tooling", runs importance weighting to MMD to PSI to TFDV, annotated "monitors the inputs". The tracks converge at 2022 on a node labelled "sequential risk monitoring". Caption: "Two lineages, one of which had the right target object for eighteen years."]

timeline
    title From detecting change to detecting harm
    1954 : Page's CUSUM establishes sequential change detection
    2000 : Shimodaira formalises covariate shift and importance weighting
    2004 : Gama et al. DDM monitors the online error rate, not the inputs
    2007 : ADWIN adapts window size with false-positive guarantees
    2012 : Gretton et al. give the kernel two-sample test and MMD
    2018 : BBSE inverts the confusion matrix to estimate label shift
    2019 : Failing Loudly finds softmax outputs are the best shift representation
         : TFX data validation runs at petabyte scale inside Google
    2021 : Accuracy-on-the-line links in-distribution and OOD performance
         : Conformal test martingales make anytime-valid change detection practical
    2022 : ATC estimates target accuracy from one confidence threshold
         : Podkopaev and Ramdas track risk with time-uniform bounds
    2024 : Sequential harmful-shift detection drops the label requirement
    2025 : Disagreement-based monitors get sample-complexity guarantees

What a Shift Detector Is Actually Computing

Strip the tooling away and almost every drift monitor is a two-sample test: reference window \(P\), current window \(Q\), a statistic, a p-value. The design space has three axes, and the literature has clear answers on two of them.

Which representation to test

Testing raw features is the default and close to the worst option. High-dimensional two-sample testing is weak, per-feature testing with a multiplicity correction discards all interactions, and the features carrying most of the variance are rarely the ones carrying the decision.

The empirical answer is to test the model's own outputs. Failing Loudly compared dimensionality reductions (no reduction, PCA, sparse random projection, autoencoders, label classifiers) crossed with univariate and multivariate tests, across many datasets and perturbation types, and found that a two-sample test on a pre-trained classifier's outputs performed best (Rabanser, Günnemann & Lipton, 2019, NeurIPS, arXiv:1810.11953). Their black-box shift detection variants, BBSDs on softmax vectors and BBSDh on hard predictions, detected shift in some settings from as few as 10 target samples. Aggregated univariate KS tests with a Bonferroni correction over the softmax dimensions beat kernel two-sample tests on the same representations in most of their comparisons, which is a pleasantly unglamorous result: the representation mattered far more than the test.

There is a reason the classifier's outputs are the right projection. The model has already learned which directions in input space change the decision. Testing its outputs is testing the input distribution weighted by relevance, which is the closest a purely distributional method gets to caring about loss.

Which statistic

Given a good representation, the statistic is second-order. KS with Bonferroni, MMD with a permutation threshold, a classifier two-sample test, and \(\chi^2\) on bins mostly differ in cost and in which alternatives they are sensitive to. PSI deserves a specific note because of how widely it is used and how badly it is understood:

with \(e_i\) and \(a_i\) the expected and actual proportions in bin \(i\). That is a symmetrised KL divergence, and it is a perfectly reasonable distance. The problem is its folklore thresholds. The bands of 0.10 for "moderate" and 0.25 for "significant, action required" were adopted in credit scoring without reference to Type I or Type II error rates, and their behaviour depends on the number of bins and the sample sizes: the 0.25 rule is roughly defensible for samples in the low hundreds with ten bins and much too conservative for larger samples (Yurdakul, 2018, Statistical Properties of Population Stability Index, Western Michigan University). Most production PSI monitors run on millions of rows with thresholds calibrated for hundreds.

How often to look, and what that does to the guarantee

This is the axis the tooling gets wrong almost universally, and the arithmetic is not subtle. Forty monitored features, one KS test each at \(\alpha = 0.05\), run every morning. Under independence, the probability that at least one fires on a stable system on any given day is

Apply Bonferroni within the day and the family-wise rate returns to 5% per day, which over a 90-day quarter gives \(1 - 0.95^{90} \approx 0.99\): on a healthy system, a near-certainty of at least one alarm per quarter, and that is the corrected version. A p-value's guarantee covers one analysis on one pre-committed sample. A monitor peeks continuously and stops when it sees something, and under that stopping rule the Type I error statement does not hold.

The fix is to change the object. A confidence sequence is an interval \((L_t, U_t)\) satisfying

The universal quantifier is inside the probability, so you may look at every timestep, stop for any reason, and coverage holds. Podkopaev and Ramdas build exactly this for a deployed model's risk, using labelled calibration data and incoming test data, and fire when the time-uniform bound crosses a pre-set tolerance (Podkopaev & Ramdas, 2022, ICLR, arXiv:2110.06177). The alarm's meaning changes with it: not "the distributions differ" but "risk exceeded the level we agreed to", which is a statement in the units of the service contract.

The machinery underneath is a nonnegative martingale under the null, bounded by Ville's inequality. Conformal test martingales apply it to the exchangeability null: conformal p-values feed a betting function whose accumulated wealth is a martingale while the data remain exchangeable, so large wealth is evidence valid at any stopping time (Vovk, 2021, Testing Randomness Online, Statistical Science 36(4), and Vovk et al., 2021, Retrain or not retrain, PMLR 152, arXiv:2102.10439). Wealth also reads better on a dashboard than a stream of p-values: it is a running total of evidence rather than a sequence of unrelated verdicts.

Estimating risk when no labels arrive

Sequential risk monitoring needs a loss to monitor, and in most production settings the loss is unavailable for weeks. Three families fill the gap, each buying its estimate with a different assumption.

Confidence thresholding. Fit a threshold \(t\) on labelled source data so that the fraction of source examples exceeding it equals source accuracy, then report the fraction of unlabelled target examples exceeding the same \(t\):

Average Thresholded Confidence does this and little else, and across WILDS, ImageNet, BREEDS, CIFAR and MNIST it estimated target performance 2 to 4 times more accurately than the prior methods it was compared against, spanning synthetic corruptions, dataset reproduction and novel subpopulations (Garg, Balakrishnan, Lipton, Neyshabur & Sedghi, 2022, ICLR, arXiv:2201.04234). What it assumes is that the relationship between confidence and correctness transfers even when the confidence distribution moves, which is weaker than assuming calibration and is why it outperforms simply averaging softmax probabilities.

Confusion-matrix inversion. Under label shift, where \(P(y)\) moves but \(P(x \mid y)\) does not, the target label marginal is recoverable in closed form by inverting the source confusion matrix against the target's predicted-label distribution, \(\hat{q}(y) = C^{-1}\mu_{\hat{y}}\) (Lipton, Wang & Smola, 2018, ICML, pp. 3122-3130, arXiv:1802.03916). The predictor may be biased or uncalibrated; it needs only an invertible \(C\). The assumption is the strong one, correct for a moving disease prevalence or fraud base rate and wrong for a new camera or an adapting attacker.

Disagreement. Where an ensemble disagrees, someone is wrong, so disagreement rate tracks error. D3M formalises this for post-deployment deterioration monitoring with sample-complexity bounds for high true-positive rates under deteriorating shift and low false-positive rates under benign shift (Nguyen et al., 2025, NeurIPS, arXiv:2506.05047). A close relative trains an explicit error estimator and feeds its predictions into the Podkopaev-Ramdas machinery as a proxy loss, giving sequential harmful-shift detection with no labels at all (Amoukou et al., 2024, NeurIPS, arXiv:2412.12910).

[IMAGE: Three-panel schematic of the estimator families. Panel 1: a confidence histogram with a vertical threshold line, source and target overlaid, shaded areas labelled "predicted accuracy". Panel 2: a 3x3 confusion matrix with an inverse symbol and a bar chart of predicted-label frequencies. Panel 3: two model decision boundaries with the wedge between them shaded and labelled "disagreement region". Caption: "Three ways to guess your error rate before the labels arrive."]

Seeing It in Motion

The architectural difference between the two designs is not which statistics get computed; both compute most of the same things. It is which one holds the pager.

flowchart TB
    subgraph L["Drift-first (common)"]
        L1["Feature windows"]:::blue
        L2["Per-feature tests<br/>daily"]:::slate
        L3["Threshold on PSI<br/>or p-value"]:::amber
        L4["Page on-call"]:::rose
        L5["Quarterly<br/>accuracy review"]:::slate
        L1 --> L2 --> L3 --> L4
    end
    subgraph R["Risk-first"]
        R1["Predictions plus<br/>confidences"]:::blue
        R2["Unlabelled risk<br/>estimate"]:::purple
        R3["Confidence sequence<br/>on risk"]:::purple
        R4["Tolerance crossed"]:::amber
        R5["Adjudicate<br/>audit sample"]:::teal
        R6["Page with a<br/>confirmed number"]:::rose
        R7["Feature tests<br/>run as diagnosis"]:::slate
        R1 --> R2 --> R3 --> R4 --> R5 --> R6
        R6 --> R7
    end
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

The feature tests survive the redesign. They move from the top of the pipeline to the bottom, from trigger to explanation, and they stop being tuned to a false-alarm budget they were never going to meet.

Over a single incident week, the loop looks like this:

sequenceDiagram
    participant S as Serving
    participant M as Risk monitor
    participant A as Audit queue
    participant O as On-call
    S->>M: predictions plus confidences, hourly
    M->>M: estimated risk 0.031, bound below tolerance
    Note over M: days 1 to 3 quiet, no page
    M->>M: estimated risk 0.052, lower bound crosses 0.045
    M->>A: sample 400 cases, stratified by score
    A-->>M: 27 confirmed errors, 6.75 percent
    M->>O: page, with estimate and audit interval
    O->>S: slice metrics by segment and version
    S-->>O: one issuer CNP segment at 19 percent error

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Watching: estimated risk rising
    Watching --> Healthy: reverts within window
    Watching --> Alarmed: time-uniform bound crosses tolerance
    Alarmed --> Confirmed: audit sample agrees
    Alarmed --> Healthy: audit sample disagrees, widen estimator
    Confirmed --> Triaged: cause attributed
    Triaged --> Remediated: rollback, patch or retrain
    Remediated --> Healthy

Watch It Run

By the Numbers

Sources: Camelyon17 figures from WILDS (Koh et al., 2021); detection sample size from Failing Loudly (Rabanser et al., 2019); estimation accuracy from ATC (Garg et al., 2022); PSI threshold provenance from Yurdakul (2018). The two alarm-rate figures are arithmetic under an independence assumption that real feature sets violate; correlated features give a somewhat lower rate, and the qualitative conclusion is unchanged.

[IMAGE: Line chart, x-axis number of monitored features from 1 to 100, y-axis probability of at least one daily false alarm. Three curves: uncorrected alpha 0.05 rising steeply past 0.9 by 50 features; Bonferroni-corrected flat at 0.05; and a dashed curve showing the Bonferroni case accumulated over 90 days, flat at 0.99. Caption: "Correcting for features does nothing about correcting for days."]

A Concrete Example

A card-fraud model, one week, two events. Both are detectable. Only one matters, and the conventional monitor gets them backwards.

Event one: the loud benign alarm. The merchant_category monitor compares this week's bucket proportions against the training baseline.

PSI \(= 0.0705 + 0.2079 = 0.2784\), above the 0.25 "action required" line. The cause: one acquirer recoded a block of grocery merchants into other on Tuesday. The same merchants, the same cardholders, the same fraud rate. The model's decisions on those transactions are byte-identical to last week's, because merchant_category is one weak feature among ninety and the recode did not move any score across the threshold. A page was raised; an engineer spent an afternoon; nothing was wrong.

Event two: the quiet harmful one. In the same week, the PSI on every feature of card-not-present traffic for one issuing bank sits at 0.03. Nothing moved. What moved is the attacker: a credential-stuffing ring changed its velocity pattern to sit just inside the model's learnt thresholds. \(P(x)\) is unchanged; \(P(y \mid x)\) is not.

The risk-first monitor sees it, in four steps.

1. Fit the threshold. On a labelled calibration set of 10,000 adjudicated transactions, accuracy is 0.972. The confidence value at which exactly 97.2% of calibration examples fall above is \(t = 0.63\).
2. Apply it to live traffic. Of this week's 120,000 unlabelled predictions, 94.8% carry confidence above 0.63. Estimated accuracy 94.8%, estimated error 5.2%, against a 2.8% baseline. Nearly double.
3. Test it sequentially. The agreed tolerance is 4.5% error. The time-uniform lower bound on risk reads 0.031 on day 1, 0.038 on day 3, 0.042 on day 4, and crosses 0.045 on day 6. On day 6 it alarms, with a guarantee that survives the fact that the team was watching the whole time.
4. Confirm against labels. 400 cases are pulled for adjudication, stratified by score. 27 come back as errors: 6.75%, with a Wald interval of roughly 4.3% to 9.2%. The estimate and the audit agree that something real is happening.

Then triage. Sliced by issuer, one bank's card-not-present segment shows 19% error while everything else sits at 2.9%. That segment is 14% of volume, so the population error should be about

5.15%, which matches the 5.2% estimate from step 2 closely enough to close the loop. Ninety-five percent of the traffic is fine and would be fine after a retrain; one segment needs a rule, a feature, or a human queue, this week.

Now run the drift monitor on that segment, which is what it is genuinely good for: it reports nothing on the raw features, and an attribution pass over covariates, the label marginal and the conditional puts the mass on the conditional (Zhang, Singh, Ghassemi & Joshi, 2023, ICML, PMLR 202:41550-41578, arXiv:2210.10769). Concept change, not covariate shift. That distinction is the difference between "collect more data from the new region" and "the adversary has learned your boundary".

[IMAGE: Two stacked time-series panels over one week. Top: PSI per feature, with one line spiking above 0.25 on Tuesday (labelled "merchant recode, benign"). Bottom: estimated error rate with a widening time-uniform lower bound, crossing the 4.5% tolerance on day 6, with the adjudicated audit point and its interval overlaid at day 6. Caption: "The loud alarm and the real one, in the same week."]

Where It Breaks

Risk monitoring inherits the label pipeline's pathologies

Everything above assumes an adjudicated sample can be obtained on demand. When labels take 18 months, as in credit default, the audit step degrades into a slow trickle and the unlabelled estimator becomes not a trigger but the metric, with no anchor. Worse, delayed labels are usually selectively observed: you learn outcomes only for applications you approved and items you showed, so the observed label set is the world filtered through the model's own policy. A model that has drifted conservative looks increasingly accurate on a shrinking population. Breaking that needs a randomised holdout or propensity weighting, both of which cost real money, and neither is optional if the monitoring signal is meant to describe the population rather than the policy.

The estimators fail in the direction that matters

All three unlabelled families are accurate under mild shift and degrade under severe shift. Confidence thresholding assumes the confidence-correctness relation transfers, and under severe shift models become overconfident in new ways. Confusion-matrix inversion assumes label shift, and the moment \(P(x \mid y)\) moves the inversion returns a confident wrong answer with no diagnostic. Disagreement assumes ensemble members err differently, which fails when they share a training set, architecture family or featurisation, and correlated members agree confidently on the same error. The pattern is consistent: they work when you did not need them.

The exchangeability null is not the null you want

Conformal test martingales are valid against a specific null, and ordinary weekly seasonality violates it. A martingale on raw production traffic will accumulate wealth and alarm, correctly rejecting exchangeability, and tell you it is Tuesday. Deseasonalise, or monitor residuals, or plan to explain the monitor to yourself weekly until someone mutes it.

Time-uniform validity costs sensitivity

A confidence sequence is wider than the fixed-sample interval at every \(t\). A genuine but small persistent degradation takes longer to cross the line, and that is the price of being allowed to look continuously. In most production settings this is the right trade. It is the wrong trade for a hard safety cut-off, which wants a deliberately over-triggering fast rule with a human behind it, and no amount of anytime-valid statistics substitutes for that.

Aggregate metrics dilute segment failures

A 2% aggregate drop is often a 30% drop in one segment that grew, or two segments moving in opposite directions and cancelling. Any monitor computed on the population inherits the mixture, and the mixture weights are themselves drifting. Slicing is not an optimisation; it is a correctness requirement, and it is also where the multiple-testing problem returns through the back door, since every new slice is another test.

Attribution only sees what you logged

A cause without a signal cannot be assigned blame, so a decomposition will distribute the entire drop across whatever is visible and give no indication that the real cause was absent. Practitioner interviews keep landing on the same conclusion: the deployed pipeline and its instrumentation, not the model, is where production ML succeeds or fails (Shankar, Garcia, Hellerstein & Parameswaran, 2022, arXiv:2209.09125). The postmortem action for an unexplained incident is an instrument, not a retrain.

Accuracy-on-the-line cuts both ways

Across CIFAR-10 and ImageNet variants, a synthetic pose task, and the FMoW and iWildCam WILDS datasets, out-of-distribution accuracy correlates strongly and often linearly with in-distribution accuracy, holding across architectures, hyperparameters, training set size and duration (Miller et al., 2021, ICML, arXiv:2107.04649). That is encouraging for model selection: the model that is better in distribution is usually better out of it. It is not licence to skip monitoring, because the correlation describes a population of models under a fixed shift, and says nothing about how far along the line any particular deployment has slid today. Later work also documents settings where the relationship weakens or inverts, so it is a regularity, not a law.

Alternative Designs

[IMAGE: A 2x2 grid positioning the seven designs. X-axis "distance from the loss", from "measures loss directly" to "measures inputs only". Y-axis "cost per week", from cheap to expensive. Labelled points: labelled audit stream top-left, sequential risk monitoring centre-left, unlabelled estimation centre, output drift centre-right, per-feature tests and PSI bottom-right, scheduled retraining off to one side. A dashed region around the left column is labelled "safe to page on". Caption: "You pay for closeness to the loss, in money or in assumptions."]

Most real stacks should run four of these: audit stream as anchor, unlabelled estimation as the continuous signal, a sequential test as the trigger, and input tests as the explanation. The mistake is running only the last one and calling it monitoring.

How It Is Used in Practice

Data validation arrived in production before drift detection did, and it remains the higher-yield investment. The TFX validation system was designed around the observation that errors in input data nullify accuracy gains, and it ran across hundreds of product teams and petabytes of daily production data (Breck et al., 2019). Most incidents that present as model degradation are schema, join or unit incidents, and a validator catches them in minutes with no statistics at all.

Regulated lending is where the drift-first design is most entrenched and most defensible, because there the convention is itself the requirement: a model risk management function expects a PSI report with the familiar bands, and producing one is compliance rather than monitoring. The practical move is to keep the report and stop paging on it.

In the open-source tooling, the split is visible. Evidently-style libraries compute the two-sample battery across features and are excellent at the diagnosis job. Performance-estimation libraries went the other way and made estimated accuracy the headline number, which is the right headline. The gap that is still closing is the sequential layer: most deployed monitors compute a statistic per window and threshold it, which is precisely the design the anytime-valid literature exists to replace.

For LLM applications the same skeleton applies with the pieces renamed. There is rarely a label; the proxy is a judge model or a rubric score, which is an error estimator with its own drift. Prompt and model version changes are a shift source the classical literature does not cover, and they are step changes rather than gradual drift, which argues for change-point framing over windowed testing. The part that transfers unchanged is the discipline: page on the score you promised someone, hold a small human-adjudicated set as the anchor, and treat everything else as explanation.

[IMAGE: Architecture diagram of a monitoring stack drawn as four horizontal layers: "audit anchor" (thin, human icon), "risk estimation" (confidence threshold, error model, disagreement), "sequential test" (a wealth process curve), and "diagnosis" (feature tests, slice metrics, attribution). Arrows show the pager attached only to the third layer. Caption: "Four layers, one pager, attached to the layer that speaks in the units of the contract."]

Insights Worth Remembering

1. Significance is not harm, and no threshold on a distributional distance converts one into the other. With enough rows every two-sample test rejects, because production input distributions always differ from a frozen baseline in some real, small way. This is not a tuning failure; it is what the test was built to do.

2. The best representation for shift detection is the model's own output. The classifier has already learned which directions change the decision, so testing softmax vectors is testing the inputs weighted by relevance. It is also the reason 10 target samples can suffice where raw-feature testing needs thousands.

3. Monitoring is a sequential problem wearing a fixed-sample costume. A p-value computed daily has no guarantee worth quoting. Confidence sequences and martingale-based tests exist precisely because the monitor never stops looking, and adopting them costs a little sensitivity and buys back the alert channel.

4. Unlabelled accuracy estimation is good enough to act on, and never good enough to act on alone. A 2 to 4 times improvement in estimation error makes it a credible trigger. It stays a trigger: resolve every alarm against adjudicated labels before changing anything.

5. The estimators degrade exactly when you need them. Confidence, inversion and disagreement all lose accuracy as shift severity rises. Design for it: alarm on sustained movement rather than a single reading, and keep the audit anchor funded.

6. Input drift is not useless, it is misplaced. Moving it from the paging path to the diagnosis path preserves everything it is good at, which is telling you what changed once something else has established that performance moved.

7. Most model incidents are data incidents. Check null rates, row counts, schema versions and join fanout before touching the model, and reproduce on a frozen replay before retraining. Retraining first destroys the evidence and can bake a pipeline bug into the weights.

Open Questions

How much of the anytime-valid machinery survives contact with non-stationary production traffic? Confidence sequences on risk are measured to hold their coverage under the assumptions they state. Production traffic has seasonality, campaign spikes and correlated arrivals, none of which are exchangeable. Whether a deseasonalisation front-end is sufficient, or whether the null needs reformulating, is largely unsettled outside the benchmarks.

Can unlabelled estimation be made reliable under severe shift, or is the degradation fundamental? The current methods share a structure: calibrate something on source data, assume it transfers. It is plausible but unproven that no purely unlabelled estimator can be uniformly accurate across shift severities, and a negative result there would usefully redirect effort toward cheaper labelling.

What is the right monitoring object for a system rather than a model? Retrieval, a reranker, a generator and a guardrail compose into an application whose risk is not any component's risk. Component-wise monitoring misses interaction failures; end-to-end monitoring cannot attribute them. Shapley-style attribution across distributions is a start, but composition across stages is not the same problem as attribution across covariates.

Does the accuracy-on-the-line regularity hold for the shifts that hurt in practice? The correlation is measured across benchmark shifts, which are mostly natural or synthetic covariate shifts. Adversarial adaptation and concept change are under-represented in those benchmark suites, and it is an open question whether the linear relationship survives them.

How should a monitor behave when the model is being updated continuously? Every fine-tune, prompt change or retrain resets the reference distribution, and a monitor whose baseline moves weekly has no long-horizon signal. Versioned baselines with explicit carry-over rules are the obvious answer and, as far as published practice goes, mostly an unsolved engineering convention problem.

Sources and Further Reading

1. Rabanser, S., Günnemann, S., & Lipton, Z. C. (2019). "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift." Advances in Neural Information Processing Systems 32 (NeurIPS). arXiv:1810.11953
2. Lipton, Z. C., Wang, Y.-X., & Smola, A. (2018). "Detecting and Correcting for Label Shift with Black Box Predictors." Proceedings of the 35^th International Conference on Machine Learning (ICML), 3122-3130. arXiv:1802.03916
3. Garg, S., Balakrishnan, S., Lipton, Z., Neyshabur, B., & Sedghi, H. (2022). "Leveraging Unlabeled Data to Predict Out-of-Distribution Performance." International Conference on Learning Representations (ICLR). arXiv:2201.04234
4. Podkopaev, A., & Ramdas, A. (2022). "Tracking the Risk of a Deployed Model and Detecting Harmful Distribution Shifts." International Conference on Learning Representations (ICLR). arXiv:2110.06177
5. Amoukou, S. I., Bewley, T., Mishra, S., Lecue, F., Magazzeni, D., & Veloso, M. (2024). "Sequential Harmful Shift Detection Without Labels." Advances in Neural Information Processing Systems 37 (NeurIPS). arXiv:2412.12910
6. Nguyen, A., et al. (2025). "Reliably Detecting Model Failures in Deployment Without Labels." Advances in Neural Information Processing Systems (NeurIPS). arXiv:2506.05047
7. Vovk, V. (2021). "Testing Randomness Online." Statistical Science, 36(4), 595-611. Project Euclid
8. Vovk, V., Petej, I., Nouretdinov, I., Ahlberg, E., Carlsson, L., & Gammerman, A. (2021). "Retrain or Not Retrain: Conformal Test Martingales for Change-Point Detection." Proceedings of Machine Learning Research, 152 (COPA). arXiv:2102.10439
9. Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). "A Kernel Two-Sample Test." Journal of Machine Learning Research, 13, 723-773. JMLR
10. Koh, P. W., Sagawa, S., Marklund, H., et al. (2021). "WILDS: A Benchmark of in-the-Wild Distribution Shifts." Proceedings of the 38^th International Conference on Machine Learning (ICML). arXiv:2012.07421
11. Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L. (2021). "Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization." Proceedings of the 38^th International Conference on Machine Learning (ICML). arXiv:2107.04649
12. Zhang, H., Singh, H., Ghassemi, M., & Joshi, S. (2023). "'Why did the Model Fail?': Attributing Model Performance Changes to Distribution Shifts." Proceedings of the 40^th International Conference on Machine Learning (ICML), PMLR 202:41550-41578. arXiv:2210.10769
13. Breck, E., Polyzotis, N., Roy, S., Whang, S., & Zinkevich, M. (2019). "Data Validation for Machine Learning." Proceedings of SysML. Google Research
14. Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. G. (2022). "Operationalizing Machine Learning: An Interview Study." arXiv:2209.09125
15. Shimodaira, H. (2000). "Improving Predictive Inference Under Covariate Shift by Weighting the Log-Likelihood Function." Journal of Statistical Planning and Inference, 90(2), 227-244.
16. Yurdakul, B. (2018). Statistical Properties of Population Stability Index. Dissertation, Western Michigan University. ScholarWorks
17. Gama, J., Medas, P., Castillo, G., & Rodrigues, P. (2004). "Learning with Drift Detection." Brazilian Symposium on Artificial Intelligence (SBIA), LNCS 3171, 286-295.
18. Bifet, A., & Gavaldà, R. (2007). "Learning from Time-Changing Data with Adaptive Windowing." Proceedings of the 2007 SIAM International Conference on Data Mining (SDM).
19. Page, E. S. (1954). "Continuous Inspection Schemes." Biometrika, 41(½), 100-115.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.