Forecast Combination and the Diversity Benefit

The most reliable accuracy improvement in forecasting is also the least intellectually satisfying: average several forecasts instead of picking one. Bates and Granger demonstrated it in 1969 on airline passenger data, showing a composite of two forecast sets could achieve lower mean squared error than either component (Bates & Granger, 1969, The Combination of Forecasts, Operational Research Quarterly 20(4), 451-468). More than fifty years and several competitions later, the finding has not been overturned; it has only been explained better.

The algebra

Take two unbiased forecasts with errors \(e_1\) and \(e_2\), variances \(\sigma_1^2, \sigma_2^2\) and correlation \(\rho\). The combined forecast \(w e_1 + (1-w) e_2\) has variance

Minimising over \(w\) gives the classic optimal weight

Two things fall out. First, the combination is at least as good as the better component whenever \(w^*\) lies in \([0, 1]\), so combining is not a compromise between good and bad, it is a variance reduction. Second, the benefit is governed by \(\rho\): two excellent but nearly identical models combine to almost nothing, while a mediocre model that is wrong in different places than the good one can still help. With \(\sigma_1 = \sigma_2 = \sigma\) and \(\rho = 0\), the equal-weight combination has variance \(\sigma^2/2\). With \(\rho = 0.9\), it has \(0.95\sigma^2\), which is a rounding error.

Diversity, not average quality, is what you are buying. This is the same decomposition that explains ensembles in supervised learning, and it is why combining an exponential smoothing model with a gradient-boosted tree usually beats combining two gradient-boosted trees with different seeds.

The combination puzzle

The optimal weights above require \(\sigma_1, \sigma_2\) and \(\rho\), which are unknown and must be estimated from a finite validation sample. Once you do that, the simple arithmetic mean repeatedly beats the estimated-optimal combination in empirical work. This is the forecast combination puzzle, and Smith and Wallis traced it to finite-sample error in estimating the weights: the variance introduced by estimating three quantities exceeds the variance saved by weighting well (Smith & Wallis, 2009, A Simple Explanation of the Forecast Combination Puzzle, Oxford Bulletin of Economics and Statistics 71(3)).

The practical implication is uncomfortable for anyone who likes optimisation. Equal weights are a strong default. If you want to weight, shrink hard towards equal weights, use long validation windows, and restrict to non-negative weights, which acts as regularisation on its own.

Evidence from the competitions

M4 was won by Smyl's hybrid of exponential smoothing and a recurrent network, itself an ensemble over many fitted models, and the organisers' headline conclusion was that combinations of methods dominated individual methods across the 100,000 series (Makridakis Competitions overview). M5 changed the flavour without changing the conclusion: the top methods were LightGBM-based, and most were combinations, including entries that mixed LightGBM with a Kalman filter, with pooled regression, and with neural networks (Makridakis, Spiliotis & Assimakopoulos, 2022, IJF 38(4)). What changed between M4 and M5 was which base models were worth combining, not whether combination helped.

When it breaks

Correlated components give nothing. Five variants of one architecture trained on one feature set are one model with noise. Measure the error correlation matrix before assuming an ensemble is diverse; a mean pairwise \(\rho\) above about 0.9 tells you the ensemble is decorative.

Quantile combination is not the same as mean combination. Averaging the 90^th percentiles of two predictive distributions does not give the 90^th percentile of the average distribution. Vincentization (averaging quantiles) and linear pooling (averaging densities) give different answers, and linear pooling of two sharp forecasts produces a wide, sometimes bimodal, distribution that is worse-calibrated than either.

Cost multiplies exactly. An ensemble of five models costs five trainings, five artefacts, five monitoring surfaces and five ways to break a deploy. The accuracy gain is often 2 to 5% on a scaled error metric, which for many decisions is not worth a fivefold increase in operational surface.

Bias does not average out. The variance argument assumes unbiased components. If every model in the ensemble uses the same leaked feature or the same stale hierarchy, the ensemble inherits the bias at full strength while looking more confident, because the spread between members has narrowed.