Online Preprocessing and Running Statistics

A scaler fitted once on last quarter's data is a constant, and by next quarter it is centring features around a mean that no longer exists. A scaler refitted on every incoming batch is worse in a subtler way: the feature map moves, so the weight the learner fitted against yesterday's scale is being applied to a differently scaled input today. Both are wrong, in opposite directions, and the choice between them is a design decision most streaming pipelines make by accident.

One-pass moments that do not explode

Computing a running mean and variance by accumulating \(\sum x\) and \(\sum x^2\) is the obvious approach and it is numerically unsound. When the mean is large relative to the spread, \(\sum x^2 - (\sum x)^2/n\) subtracts two nearly equal large numbers and the result is dominated by rounding error; sensor readings near \(10^9\) with unit spread lose the variance entirely in single precision.

Welford's recurrence avoids the cancellation by updating around the current mean:

with the sample variance read off as \(S_k/(k-1)\) (Welford, 1962, Note on a Method for Calculating Corrected Sums of Squares and Products, Technometrics 4(3):419-420). Two values of state, one pass, stable across scales. It is the default in every serious streaming library and in most numerical packages.

Forgetting, and the rate that matters

A running mean over all history is the wrong statistic for a drifting stream, so the usual substitute is exponential weighting:

with the variance tracked the same way. The effective window is about \(1/\lambda\) observations, which makes \(\lambda\) an interpretable knob. Fading factors like this are the preferred mechanism over explicit sliding windows for streaming estimation because they are memoryless and cheaper, while converging to the same quantity on stationary data (Gama, Sebastião and Rodrigues, 2013, On Evaluating Stream Learning Algorithms, Machine Learning).

Now the coupling problem. If \(\mu_t\) and \(\sigma_t\) move, then the standardised feature \(z_j = (x_j - \mu_j)/\sigma_j\) means something different at each step, and the weight \(w_j\) the learner holds was fitted against earlier values of it. Two designs are defensible. Freeze the transform and let the learner absorb the drift in its weights, which keeps the feature map interpretable and puts all the adaptation in one place. Or let the transform move much more slowly than the learner adapts, so the learner tracks the moving scale as just another slow drift. Streaming libraries take the second route by default (Montiel et al., 2021, River: Machine Learning for Streaming Data in Python, JMLR 22(110):1-8), and the thing to know is that it makes the model non-stationary in a way no model metric reports. What you must not do is move the transform at the learner's own timescale, which is what naive per-batch refitting does.

Predict, then learn

The ordering is not a style preference. Statistics must be updated with \(x_t\) only after the prediction for \(x_t\) has been emitted. Fold \(x_t\) into the running mean first and the example has helped normalise itself, and prequential accuracy comes out optimistically biased by an amount nobody measures.

For target-derived transforms the leak is direct rather than subtle. Encoding a category by its mean label is a well-known leak in batch pipelines, handled there by cross-fitting inside the encoder. A stream's equivalent is strictly causal: encode \(x_t\) using only the counts and label sums accumulated before \(t\), update afterwards. Unseen categories need a defined fallback, usually the global running mean with a shrinkage weight based on the category count, because a category seen twice should not be trusted as much as one seen ten thousand times.

When it breaks

A single outlier captures the variance. One reading of \(10^{12}\) inflates \(\sigma\) for as long as the forgetting factor remembers it, and every subsequent feature is squashed toward zero. Clipping before the update, or scaling by an interquartile range from a quantile sketch, costs more and survives heavy tails.

Regime change leaves the scaler straddling two worlds. After a step change the running mean is a blend of the old and new regimes for roughly \(1/\lambda\) observations, so features are mis-scaled throughout the period when the model is also adapting. The two adaptations interact, and the usual symptom is an error spike considerably longer than the drift itself.

Early estimates are noise. The first few dozen examples give a variance estimate that is mostly sampling error, so an online model's earliest predictions are normalised by something close to random. A warm-up during which the transform is held at a prior, or predictions are suppressed, is the standard fix and is routinely skipped.

Nothing monitors the transform. Dashboards watch model accuracy and input distributions; almost none watch the scaler's own parameters. A drifting \(\mu_j\) is invisible until it is large enough to show up as model degradation, by which point the cause is several steps removed from the symptom.