Sequential monitoring and always-valid alarms

A team runs a Kolmogorov-Smirnov test on 40 monitored features every morning at \(\alpha = 0.05\). Assuming independence, the probability that at least one test fires on a stable system on a given day is \(1 - 0.95^{40} \approx 87\%\). Over a quarter, effectively every day produces an alert, the channel is muted within a fortnight, and the monitoring system's actual false-negative rate becomes 100% because nobody is reading it.

This is not a threshold-tuning problem. It is the structural mismatch between fixed-sample statistics and a monitor that never stops looking.

Why the classical guarantee does not apply

A p-value's Type I error guarantee is a statement about one analysis on one pre-committed sample. A monitor peeks continuously, on growing data, and stops when it sees something. Under that stopping rule, \(P(\exists t: p_t < \alpha)\) is not \(\alpha\); for a statistic that keeps accumulating, it approaches 1. The same arithmetic that makes A/B test peeking invalid makes daily drift testing invalid, with the added problem that monitoring runs forever while an experiment ends.

Two corrections are usually applied and neither fixes it. Bonferroni across features controls the family-wise rate within one day and does nothing about the sequence of days. Raising the threshold trades a still-uncontrolled false alarm rate for lost sensitivity to the shifts that matter.

Time-uniform bounds

The correct object is a confidence sequence: an interval \((L_t, U_t)\) constructed so that

The quantifier moved inside the probability. You may look at every \(t\), stop whenever you like, and the coverage still holds. This is what makes a monitor statistically legitimate rather than a machine for generating significant-looking noise.

Applied to deployed models, risk monitoring builds a time-uniform lower bound on the model's current risk from labelled calibration data and incoming test data, and alarms when the bound exceeds a pre-set tolerance (Podkopaev & Ramdas, 2022, ICLR, arXiv:2110.06177). Crucially, it alarms on harmful shift, defined by the risk crossing a threshold, not on distributional difference. The label-free extension replaces the observed loss with an error-estimator proxy and keeps the same sequential machinery (Amoukou et al., 2024, NeurIPS, arXiv:2412.12910).

Martingales and betting

The underlying mechanism is a nonnegative martingale under the null, so that Ville's inequality bounds the probability it ever exceeds \(1/\alpha\). Conformal test martingales instantiate this for the exchangeability null: conformal p-values are fed into a betting function whose accumulated wealth is a martingale as long as data remain exchangeable, and large wealth is evidence against exchangeability that is valid at any stopping time (Vovk, 2021, Testing Randomness Online, Statistical Science 36(4)). The wealth process is also more interpretable than a p-value stream: it is a running log of how much evidence has accumulated, and it is readable on a dashboard.

When it breaks

Time-uniform validity is not free; it is paid for in sensitivity, and a confidence sequence is wider than the fixed-sample interval at every \(t\). A real, small, persistent degradation takes longer to reach the alarm line. This is the correct trade in production, where a 2% accuracy loss discovered in six days beats a false alarm every day, but it does mean sequential monitoring is the wrong tool for a hard real-time safety cut-off, which needs a fast, deliberately over-triggering rule with a human behind it. The other limit is the null itself: exchangeability fails under ordinary seasonality, so a conformal martingale on raw traffic will detect Tuesday. Monitor residuals or deseasonalised quantities, or expect to explain your alarms to yourself weekly.