Gradient-Boosted Trees for Forecasting

In the M5 competition, 30,490 Walmart product-store series were forecast 28 days ahead by 5,507 teams. Every one of the top five methods used LightGBM, and for the first time in the history of the M competitions all top-performing entries were pure machine learning and beat every statistical benchmark and their combinations (Makridakis, Spiliotis & Assimakopoulos, 2022, M5 accuracy competition: Results, findings, and conclusions, IJF 38(4)). Two years earlier, in M4, the winner had been a hybrid of exponential smoothing and an RNN. Something changed, and it was not the tree algorithm, which was already old news.

The reframing is the whole trick

A gradient-boosted tree has no notion of time. It sees a table. Forecasting becomes tabular regression the moment you build one row per (series, time origin, horizon) with the target \(y_{t+h}\) and features drawn only from information available at \(t\):

• lags: \(y_{t}, y_{t-1}, y_{t-7}, y_{t-28}, y_{t-364}\), chosen to cover the seasonal periods that exist in the domain
• rolling aggregates: mean, standard deviation, share of zero days over trailing windows of 7, 28 and 180 days
• calendar: day of week, day of month, week of year, holiday flags, days-to-event and days-since-event
• static attributes: department, category, store, region, price tier
• price and promotion: current price, price relative to a trailing median, promotional flags

The tree then learns interactions between them without being told they exist, which is where its advantage over a linear global model lies. "Snap benefit day, in Texas, for a food item under $3" is a four-way interaction that a tree finds by splitting and a linear model finds only if you write the term yourself.

This is not a competition curiosity. Elsayed and colleagues took a plain gradient-boosted regression tree, gave it the same window-based feature transformation, and found it outperformed a set of published deep forecasting architectures on their own benchmark datasets, arguing that the reported progress of deep models was partly an artefact of weakly configured baselines (Elsayed et al., 2021, Do We Really Need Deep Learning Models for Time Series Forecasting?, arXiv:2101.02118).

Global by construction

One table holding every series means one model fitted across all of them, which is the real source of the accuracy. A new store with six weeks of history inherits everything the model learned about day-of-week shape from 30,000 other series. The tree does not know which rows came from which series unless you give it a series identifier, and mostly you should not: the static attributes carry the useful part of the identity and generalise to series the model has never seen.

Scaling matters more here than in a neural global model. Trees split on raw feature values, so a series selling 5,000 units a day and one selling 3 cannot share a threshold usefully. The standard fixes are to model a scaled target, \(y / \bar{y}_{\text{trailing}}\), or a log-ratio, and to include the scale itself as a feature so the model can condition on it.

Horizon handling

Two conventions dominate. One model per horizon (direct) gives each \(h\) its own feature set, using only lags of at least \(h\), which is clean but multiplies training cost by \(H\). One model with \(h\) as a feature trains a single model over rows that carry the horizon as an input column, which shares strength across horizons and is what most M5 solutions did in some form. Recursive use of a tree is possible and mostly discouraged: the lag features have to be rebuilt from predictions at every step, and the error compounds through a model that was never trained on its own outputs.

When it breaks

Trees cannot extrapolate. A regression tree predicts the mean of a leaf, so its output is bounded by the range of targets in training. A series in genuine exponential growth will be forecast flat at the top of its observed range, and no amount of boosting fixes it. The standard workaround is to model differences or ratios rather than levels, which moves the extrapolation into a transform the tree does not have to learn.

Long horizons starve the feature set. At \(h = 90\), every lag shorter than 90 is unavailable, so the model loses exactly the recent-history features that carry most of the signal at short horizons. Accuracy degrades faster with horizon than for models that carry state.

Feature engineering is the model. The architecture is fixed; what varies is the table. That is an advantage for iteration speed and a liability for reproducibility, because the interesting decisions live in preprocessing code rather than in a config file, and leakage hides there. A rolling mean computed over a window that includes \(t+1\) will not fail any test, it will just look brilliant in backtest.

Probabilistic output needs extra work. A point-forecast tree gives no distribution. The usual answer is to fit one model per quantile with pinball loss, which multiplies training cost by the number of quantiles and does not enforce monotonicity between them, so quantile crossing has to be repaired afterwards.

Cold start still needs a story. A brand new SKU has no lags at all. The global model degrades gracefully because static attributes still carry information, but the lag features are imputed, and how you impute them is a modelling decision that deserves its own evaluation.