Trees & Ensembles
Impurity splitting, bagging, random forests, gradient boosting, and the engineering inside XGBoost and LightGBM.
7concepts
145flashcards
56minutes of reading
- 01 Bagging and Random Forests Why averaging unstable models reduces variance, why bootstrap sampling alone is not enough, and what the extra feature subsampling in a random forest is actually buying.
- 02 Stacking and Blending Stacking learns how to combine several models from their out-of-fold predictions, and nearly all of its value and all of its danger come from how those predictions are produced.
- 03 Why Trees Still Beat Deep Nets on Tabular Data The three inductive biases that separate tree ensembles from neural networks on tabular problems, and the specific dataset conditions under which the ordering has been observed to flip.