Forecast Evaluation at Scale

It is undefined at zero and explodes at small actuals, which is exactly the intermittent-demand regime where forecasting is hardest. And it is asymmetric: over-forecasting is unbounded while under-forecasting is capped at 100 percent, so minimising it systematically biases toward under-forecasting, invisibly and expensively.

The in-sample mean absolute error of a naive forecast, typically seasonal naive. Below one means better than that benchmark, above one means worse. It is scale-free, defined whenever the naive forecast has non-zero error, and interpretable, which is why it is the recommended default for cross-series comparison.

When business cost is proportional to units, since it aggregates total absolute error over total actuals and thus weights by volume. It deliberately ignores small series, so it belongs alongside a scale-free metric rather than instead of one, or the long tail becomes unmeasured.

Because a single number over a heterogeneous population is dominated by the most numerous or largest series, so excellent performance on high-volume smooth series and useless performance on the intermittent tail reports a good average. The tail is usually where inventory cost concentrates.

Stratification: by volume decile, by intermittency, by series age, and by whatever segmentation the business decisions follow. The distribution of per-series MASE, particularly its upper tail, is more informative than its mean and is what identifies which population the model fails on.

Because a single train-test split gives one noisy estimate and random splits leak the future, while rolling origin refits and forecasts forward repeatedly to give a distribution of errors over time. It is skipped because it costs many refits, which for a large global model is a substantial compute bill.

Because the decision applies a cost structure to the forecast, so errors in the expensive direction matter more than their magnitude suggests. A model with lower average error whose residuals are biased toward the costly side is worse for the business, which is why evaluating on decision cost is closer to the thing that matters.

It is undefined at zero and explodes at small actuals, which is exactly the intermittent-demand regime where forecasting is hardest. And it is asymmetric: over-forecasting is unbounded while under-forecasting is capped at 100 percent, so minimising it systematically biases toward under-forecasting, invisibly and expensively.

The in-sample mean absolute error of a naive forecast, typically seasonal naive. Below one means better than that benchmark, above one means worse. It is scale-free, defined whenever the naive forecast has non-zero error, and interpretable, which is why it is the recommended default for cross-series comparison.

When business cost is proportional to units, since it aggregates total absolute error over total actuals and thus weights by volume. It deliberately ignores small series, so it belongs alongside a scale-free metric rather than instead of one, or the long tail becomes unmeasured.

Because a single number over a heterogeneous population is dominated by the most numerous or largest series, so excellent performance on high-volume smooth series and useless performance on the intermittent tail reports a good average. The tail is usually where inventory cost concentrates.

Stratification: by volume decile, by intermittency, by series age, and by whatever segmentation the business decisions follow. The distribution of per-series MASE, particularly its upper tail, is more informative than its mean and is what identifies which population the model fails on.

Because a single train-test split gives one noisy estimate and random splits leak the future, while rolling origin refits and forecasts forward repeatedly to give a distribution of errors over time. It is skipped because it costs many refits, which for a large global model is a substantial compute bill.

Because the decision applies a cost structure to the forecast, so errors in the expensive direction matter more than their magnitude suggests. A model with lower average error whose residuals are biased toward the costly side is worse for the business, which is why evaluating on decision cost is closer to the thing that matters.