Prequential Evaluation
Why a stream has no test set, how test-then-train uses every example twice without leaking, and the forgetting mechanisms that make the resulting accuracy curve informative.
A stream has no natural train-test split. Holding out the last portion tests on one period and wastes it for training; holding out at random breaks temporal order and leaks the future. The evaluation protocol that fits the setting uses every example for both purposes, in an order that makes leakage impossible.
Test-then-train
For each arriving example: predict, record the loss against the true label, then update the model on it. Every example is a test case before it is a training case, so the recorded loss is always an honest out-of-sample measurement, and no data is set aside.
The accumulated loss is the prequential error, and it has an appealing interpretation: it is the total loss the model actually incurred while operating, which is precisely what a deployed system's cost is. There is no held-out estimate standing in for deployment performance, because the measurement is taken during deployment.
Why plain accumulation is not enough
The cumulative average over the whole stream is dominated by the early period when the model was poor, so it understates current performance indefinitely and moves more slowly as the stream grows. After a million examples, a change in behaviour barely shifts the running mean.
Two corrections are standard.
A sliding window over the last \(n\) predictions reports recent performance directly. Simple, interpretable, and the window length is a parameter that determines how quickly the metric responds and how noisy it is.
Fading factors apply exponential decay to older losses, giving a smooth estimate with one parameter and no stored window, which is preferable when memory matters or when the response should be graded rather than sharp.
Either makes the accuracy curve informative about the present, which is the whole point: a prequential curve that dips at a drift point and recovers is a direct picture of how well the adaptation worked.
When it breaks
Delayed labels break the protocol. Test-then-train assumes the label is available before the next example arrives. With labels arriving hours later, the model is updated on examples whose predictions were recorded long ago, and the evaluation and the training run on different clocks. The protocol has to be adapted to the actual label latency rather than assumed away.
A single stream is a single sample. The curve describes what happened on this sequence, and a different arrival order would give a different curve. Comparing two algorithms on one stream is comparing them on one draw, so repeated runs over resampled or multiple streams are needed for any claim about which is better.
Adaptation and accuracy are confounded. A model that adapts fast recovers quickly from drift and tracks noise between drifts, so its prequential curve can look better or worse depending entirely on how much drift the stream happened to contain. Reporting the drift structure alongside the curve is what makes it interpretable.
Class imbalance drifts too. Accuracy on a stream whose class balance changes moves for reasons unrelated to the model, so prequential accuracy on imbalanced streams should be paired with a metric that is invariant to prevalence, such as a per-class recall or a balanced accuracy.
12 flashcards for this concept
Click a card to reveal the answer.