Deep Forecasting Architectures: DeepAR, N-BEATS and TFT

In the M4 competition of 2018, the six pure machine learning entries ranked 23^rd, 37^th, 38^th, 48^th, 54^th and 57^th out of 60, and the best-ranking methods were mostly ensembles of classical statistical techniques. That tally was quoted as the motivation for N-BEATS (Oreshkin et al., 2020, N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, ICLR, arXiv:1905.10437), and it framed the question every deep forecasting paper since has had to answer: better than what, measured how? DeepAR, N-BEATS and the Temporal Fusion Transformer each gave a different architectural answer, and the differences are more instructive than the benchmark numbers.

DeepAR: a global autoregressive likelihood

DeepAR trains one recurrent network across many related series (Salinas et al., 2020, DeepAR: Probabilistic forecasting with autoregressive recurrent networks, IJF 36(3)). At each step the LSTM state \(h_{i,t} = h(h_{i,t-1}, z_{i,t-1}, x_{i,t})\) consumes the previous value \(z_{i,t-1}\) and covariates \(x_{i,t}\), and emits the parameters \(\theta_{i,t}\) of a likelihood \(\ell(z_{i,t} \mid \theta_{i,t})\). Training maximises the log-likelihood of observed values. For counts, the paper uses a negative binomial with mean \(\mu\) and shape \(\alpha\), so that \(\mathrm{Var}[z] = \mu + \mu^2\alpha\), which handles overdispersed demand directly.

Forecasts come from ancestral sampling: draw \(\tilde{z}_{t}\) from the predicted distribution, feed it back as the next input, and repeat to the horizon. Many sample paths give joint predictive distributions, so quantities such as total demand over a lead time can be read off directly, which marginal quantile forecasts cannot provide.

The detail that made it work at Amazon scale was scale handling. Sales velocities across items followed an approximate power law, so the network divides inputs by a per-series factor \(\nu_i = 1 + \frac{1}{t_0}\sum_{t=1}^{t_0} z_{i,t}\) and multiplies the output mean back. The paper reports accuracy improvements of around 15% over the methods it compared against.

N-BEATS: depth without time-series machinery

N-BEATS is a deep stack of fully connected blocks with no recurrence, no attention and no covariates. Each block receives a lookback window of 2H to 7H points for horizon H and emits two outputs: a backcast, its reconstruction of the input, and a partial forecast. The next block receives the input minus the backcast, so each block models what the previous ones could not explain, and the final forecast is the sum of all partial forecasts. The authors call this doubly residual stacking.

Constraining the final layer of each block to a polynomial basis in one stack and a Fourier basis in another yields an interpretable variant whose outputs decompose into trend and seasonality. The generic variant uses learned bases and, as the authors put it, contains no time-series-specific components at all.

On M4, N-BEATS reported 11% better accuracy than a statistical benchmark and 3% better than the M4 winner, a hybrid of exponential smoothing and recurrent networks. The number carries a cost that often goes unmentioned: the result is an ensemble of 180 models trained on three losses, six lookback lengths and multiple seeds, each taking 30 minutes to 2 hours of GPU training on M4. That is on the order of 90 to 360 GPU-hours before any tuning. An ablation shows 18 models already reach competitive accuracy.

TFT: typing the inputs

The Temporal Fusion Transformer starts from a practical observation: real forecasting inputs come in kinds (Lim et al., 2021, Temporal Fusion Transformers for interpretable multi-horizon time series forecasting, IJF 37(4)). Static metadata (store, category), inputs known in advance (calendar, planned promotions) and inputs observed only historically (past sales, weather) should be processed differently. TFT uses static covariate encoders to condition the rest of the network, variable selection networks that weight inputs per time step, gated residual blocks that let the network skip unneeded components, an LSTM sequence-to-sequence layer for local patterns, and interpretable multi-head attention for long-range dependencies.

It is a direct multi-horizon model: all horizons are predicted at once as quantiles, trained with quantile loss, so there is no sampling feedback loop. The variable selection weights and attention patterns give feature importance and seasonal structure as a by-product, which is often why teams choose it.

Where the field disagrees

The sharpest challenge came from long-horizon benchmarks. Zeng et al. compared Transformer-based long-term forecasting models with a single linear layer and reported that the linear models outperformed the Transformers on all nine datasets, often by a large margin (Zeng et al., 2023, Are Transformers Effective for Time Series Forecasting?, AAAI, arXiv:2205.13504). Their argument is that permutation-invariant attention loses the ordering information forecasting depends on. The counter-position, taken by later patch-based and foundation models, is that the architectures were poorly adapted to time series rather than unsuitable. The settled part is methodological: a deep model reported without seasonal naive, tuned exponential smoothing and a linear or gradient-boosted baseline has not shown anything yet.

When it breaks

Autoregressive sampling compounds error. DeepAR is trained on true previous values and forecasts on its own samples, so early mistakes feed later steps. Direct models like TFT avoid that but must learn every horizon separately.

Covariates are uneven. Original N-BEATS is univariate. When promotions or prices drive demand, a model that cannot see them loses to a gradient-boosted tree that can, as M5 illustrated.

The ensemble is the method. Reported accuracy often depends on ensembling and tuning budgets that dwarf the baselines', and comparisons at equal compute are rare.

Few, short series favour classical methods. These are global models that need many related series; on a handful of short histories, per-series statistical models are frequently better, as discussed under global versus local models.