Online Conformal Prediction Under Drift

Split conformal prediction is the standard way to turn a point predictor into a calibrated interval, and it rests on exchangeability: the calibration data and the test point must be interchangeable in distribution. A drifting stream violates that by construction. Keep calibrating on a trailing window and coverage silently drops below the nominal rate exactly when the model is least reliable, which is the moment the interval was supposed to protect you.

Adaptive conformal inference fixes this by giving up on validity per step and controlling the error rate over time instead.

The miscoverage level as a controlled variable

Fix a target miscoverage \(\alpha\), say 0.1 for a 90 percent interval. Rather than always taking the \((1-\alpha)\) quantile of recent conformity scores, maintain a time-varying level \(\alpha_t\) and build the set at level \(1 - \alpha_t\). After each observation, record whether the true value fell outside the set, \(\text{err}_t \in \{0, 1\}\), and update with step size \(\gamma > 0\):

A miss pushes \(\alpha_t\) down, which raises the quantile and widens the next set. A hit nudges \(\alpha_t\) up and narrows it. This is an integral controller on the coverage error, and the resulting bound is a telescoping sum rather than a probabilistic argument:

That is Proposition 4.1 of Gibbs and Candès, 2021, Adaptive Conformal Inference Under Distribution Shift, NeurIPS 34, arXiv:2106.00170. It holds for any data-generating process, including an adversarial one, because it never assumes anything about the data. The proof only needs the recursion and the fact that \(\alpha_t\) stays within \([-\gamma, 1+\gamma]\).

The asymmetry is the mechanism

The two cases are not symmetric, and the ratio is what makes the controller work. With \(\alpha = 0.1\) and \(\gamma = 0.05\):

• covered: \(\alpha_{t+1} = \alpha_t + 0.05(0.1 - 0) = \alpha_t + 0.005\)
• missed: \(\alpha_{t+1} = \alpha_t + 0.05(0.1 - 1) = \alpha_t - 0.045\)

One miss undoes nine hits, matching the \((1-\alpha)/\alpha\) ratio the target implies. Under a shock, a run of misses drives \(\alpha_t\) to zero or below, at which point the prescribed quantile exceeds every observed score and the set becomes the whole real line. An infinite interval is useless and it is also the only honest output available: the system is announcing that it cannot bound the error right now. Coverage is maintained, in the arithmetic sense, by refusing to commit.

Tuning the step size, and what replaced hand-tuning

\(\gamma\) is the whole configuration and it is a genuine tradeoff. Small \(\gamma\) adapts slowly, so a regime change produces a long stretch of under-coverage. Large \(\gamma\) tracks quickly and makes widths jump around, which users read as instability. The follow-up work runs a set of candidate step sizes as experts and aggregates them online, so no single \(\gamma\) has to be chosen in advance (Gibbs and Candès, 2024, Conformal Inference for Online Prediction with Arbitrary Distribution Shifts, JMLR 25(162):1-36, arXiv:2208.08401).

Conformal PID goes further and treats the whole thing explicitly as control: the integral term above, a proportional term on the current score, and a forecast of the score itself for the derivative-like component. On 4-week-ahead COVID-19 death forecasting it improved coverage relative to the ensemble the CDC was publishing (Angelopoulos, Candès and Tibshirani, 2023, Conformal PID Control for Time Series Prediction, NeurIPS, arXiv:2307.16895).

When it breaks

Long-run average coverage is a weak promise. The bound constrains the time average over \(T\) steps. It permits long stretches of systematic under-coverage followed by compensating over-coverage, and both stretches can be exactly the periods a user cares about. Nothing here delivers conditional coverage for a subgroup, a time window, or a particular input region.

Infinite sets count as covered. A system that outputs the real line for a thousand consecutive steps satisfies the guarantee perfectly. Coverage is therefore not a sufficient monitor: average width, the fraction of steps with an unbounded or implausibly wide set, and the distribution of widths all have to be tracked alongside it.

Labels must arrive fast. The recursion needs \(\text{err}_t\) before step \(t+1\). Under delayed labels the controller is acting on stale feedback, the effective loop gain changes, and the stated bound no longer describes what is running. Variants that handle a fixed delay exist, but the delay has to be modelled rather than ignored.

The underlying model still has to be decent. Conformal methods calibrate an interval around whatever predictor they wrap. A badly drifting point predictor yields wide, truthful, useless intervals, and the conformal layer will not tell you to retrain. That signal has to come from somewhere else, such as a drift detector on the model's own error.