Backtesting and Temporal Validation

A demand model scores 0.94 \(R^2\) in five-fold cross-validation and 0.31 in production. The folds were random, so for a January test point the model trained on February and March. It learned the level, the seasonality and the specific shocks of the period it was being tested on. Every number in that evaluation was arithmetically correct and answered a question nobody asked.

Random splitting assumes exchangeability, and a time series is defined by its ordering. Once time matters, the only honest evaluation trains on the past and tests on the future.

The three schemes

Rolling origin with an expanding window. Train on everything up to \(t\), forecast \(h\) steps, advance \(t\), repeat. This matches production for a model retrained on all available history, and it produces many evaluation points from one series. Early folds are trained on much less data than later ones, so the aggregate score mixes regimes of very different training-set size.

Rolling origin with a sliding window. Same, with a fixed-length training window that drops the oldest data. This matches production for a model that deliberately forgets, and it keeps folds comparable. It discards history that may still be informative.

A single holdout at the end. The most honest for a one-shot decision and the highest-variance, since the result depends entirely on what happened to occur in that final period. A seasonal peak or a one-off event in the holdout can dominate the conclusion.

The choice should mirror the deployment. If the model is retrained monthly on everything, evaluate with an expanding window and monthly origins. Evaluating in a way the system will never operate is a category error, however statistically tidy it looks.

The details that decide validity

The gap. If labels arrive with a delay, the model at time \(t\) does not have data up to \(t\). A purge period between train and test, sized to the label delay, is required for the evaluation to be achievable. Without it the backtest assumes information the system will not have.

Feature computation windows. A rolling 30-day average computed over the full dataset before splitting leaks future values into training rows. Every derived feature must be computed within the training window, which usually means computing features inside the fold rather than once up front.

Preprocessing. Scalers, encoders and imputers fitted on the full series carry future information. This is the same point as feature computation and is missed just as often, because a scaler feels like configuration rather than a fitted model.

Horizon consistency. A model evaluated at one-step-ahead and deployed at 30-steps-ahead has not been evaluated. Error grows with horizon, usually sharply, and the correct backtest forecasts exactly the horizon that will be used.

Reading the result

Aggregate error across folds hides the thing that usually matters: whether performance is stable over time. Plotting error per origin frequently shows a model that was excellent for two years and degraded after a regime change, which no single average reveals.

Scale-dependent metrics such as RMSE cannot be averaged across series with different magnitudes; MASE, which scales by the in-sample error of a naive forecast, is the standard for that case and has the useful property that a value above 1 means the model is worse than the naive baseline. sMAPE is common and behaves badly near zero, which matters for intermittent demand.

When it breaks

Multiple series need grouped temporal splitting. With many related series, the split must be by time across all of them, not per series independently, otherwise one series' future informs another's past through shared parameters.

Backtesting many configurations overfits the backtest. Fifty model variants evaluated on the same rolling origin, with the best selected, gives a selected score that is optimistic by exactly the selection. A final untouched holdout after selection is the only defence.

Recursive multi-step forecasts compound their own errors. Feeding a model's prediction back in as an input for the next step means backtest errors accumulate the way production errors will, which is correct. Direct multi-step models, one per horizon, avoid this and multiply the training cost.

A backtest cannot see interventions. Historical data contains the effects of past pricing decisions, stockouts and promotions. A model that scores well by learning those patterns is forecasting a world in which the same decisions keep being made, which is a conditional forecast presented as an unconditional one.