Stacking and Blending

Two models have prediction errors with standard deviations 1.0 and 1.2. If their errors were correlated at 0.3, the best weighted average would cut the error standard deviation to about 0.87, a 13% gain over the better model for free. At a correlation of 0.8, the best average is about 0.998, which is no gain at all. Combining models is only as good as their disagreement, and stacking is the systematic way of learning a combination while not being fooled about how much the models actually disagree on new data.

The arithmetic of combining two models

For unbiased predictors with error variances \(\sigma_1^2, \sigma_2^2\) and error correlation \(\rho\), the weight on model 1 that minimises the variance of \(w\hat y_1 + (1-w)\hat y_2\) is

With \(\sigma_1 = 1\), \(\sigma_2 = 1.2\): at \(\rho = 0.3\), \(w^* \approx 0.63\) and the combined variance is about \(0.76\); at \(\rho = 0.8\), \(w^* \approx 0.92\) and the variance is about \(0.997\). Two practical lessons fall out. Diversity is worth more than the individual quality of a second model, and a combiner that sees in-sample predictions will badly misjudge \(\rho\), because every model fits its own training data almost perfectly and the errors look small and uncorrelated.

Out-of-fold predictions are the mechanism

Wolpert's stacked generalisation used a second-level learner trained on the outputs of first-level learners, with those outputs produced on data the first level had not been trained on (Wolpert, 1992, Stacked Generalization, Neural Networks 5(2)). The standard modern recipe is:

1. Split the training data into \(K\) folds.
2. For each base model \(m\) and fold \(k\), train on the other folds and predict fold \(k\), producing an out-of-fold prediction \(z_{im}\) for every row \(i\).
3. Train a meta-learner on the matrix \(Z = [z_{im}]\) with the original targets.
4. Refit each base model on all the data; at prediction time, feed their outputs to the meta-learner.

Each out-of-fold \(z_{im}\) is an honest sample of how model \(m\) behaves on unseen data, so the meta-learner learns weights from realistic errors and correlations. The cost is \(K\) fits per base model plus one refit: five base models with 5 folds is 30 fits.

Breiman found that the meta-learner should be simple and constrained. Stacking regression trees of different sizes and linear subset regressions, he combined cross-validated predictions by least squares with non-negativity constraints on the weights and found the constraint mattered for good performance (Breiman, 1996, Stacked Regressions, Machine Learning 24). Unconstrained least squares on highly collinear predictions produces large offsetting weights of opposite sign that do not generalise.

Blending, super learners, and multi-layer stacks

Blending replaces the \(K\)-fold loop with a single holdout: base models train on one part, predict the holdout, and the combiner trains on those holdout predictions. It is simpler, avoids refitting, and cannot leak across folds, at the cost of training base models on less data and fitting the combiner on a small sample. The name comes from competition practice rather than from a single paper, and the terms are used inconsistently.

The super learner gives stacking a theoretical footing: using V-fold cross-validation to choose a weighted combination of candidates, van der Laan, Polley and Hubbard showed the combination performs asymptotically as well as the best weighted combination of the candidates, the oracle (van der Laan, Polley & Hubbard, 2007, Super Learner, Statistical Applications in Genetics and Molecular Biology 6(1)).

AutoML systems pushed the idea further. AutoGluon-Tabular stacks models in multiple layers, feeding each layer's out-of-fold predictions together with the original features to the next, and uses repeated \(k\)-fold bagging to reduce variance. Its authors argue that combining many models this way uses a time budget better than searching for a single best model (Erickson et al., 2020, AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data, arXiv:2003.06505).

This is where opinions split. Competition and AutoML results reward deep stacks for small metric gains. Many production teams refuse them: a 0.3% AUC improvement from a stack of 20 models is rarely worth 20 models' worth of feature pipelines, monitoring, latency and retraining, and a single well-tuned gradient-boosted model is the common choice.

When it breaks

Leakage through in-fold predictions. Training the meta-learner on predictions the base models made on their own training rows is the classic failure. The meta-learner trusts whichever model overfit most, and the stack underperforms its best component in production.

Folds must match the data's structure. Out-of-fold predictions are only honest if the folds are. Grouped entities split across folds, or random folds on time series, leak through the first level even when the second level is done correctly. Every base model must also use the same fold assignment, or a row's predictions come from models that saw it.

Target encodings and early stopping leak too. A base model whose preprocessing, such as target encoding, or early-stopping round was fitted using the out-of-fold rows produces optimistic \(z_{im}\). Everything data-dependent must live inside each fold.

The refit changes the models. Base models refit on all data are slightly different from the fold models that generated the meta-features, and models that vary a lot with the training sample, such as deep unregularised trees, shift the most. Averaging the \(K\) fold models instead of refitting avoids the mismatch at \(K\) times the inference cost.

Operational cost scales with the stack. Every base model is a dependency that can drift, fail or slow down, and diagnosing a bad prediction means inspecting several models and a combiner.