Trees & Ensembles
Impurity splitting, bagging, random forests, gradient boosting, and the engineering inside XGBoost and LightGBM.
7concepts
145flashcards
56minutes of reading
- 01 Decision Trees and Impurity Splitting How a greedy search over axis-aligned splits builds a piecewise-constant function, why Gini and entropy almost never disagree, and the specific structural biases that make a single tree unstable.
- 02 Bagging and Random Forests Why averaging unstable models reduces variance, why bootstrap sampling alone is not enough, and what the extra feature subsampling in a random forest is actually buying.
- 03 Stacking and Blending Stacking learns how to combine several models from their out-of-fold predictions, and nearly all of its value and all of its danger come from how those predictions are produced.
- 04 Why Trees Still Beat Deep Nets on Tabular Data The three inductive biases that separate tree ensembles from neural networks on tabular problems, and the specific dataset conditions under which the ordering has been observed to flip.
- 05 Gradient Boosting as Functional Gradient Descent Boosting is gradient descent in function space, where each new tree approximates the negative gradient of the loss, which is what lets a single algorithm fit any differentiable objective.
- 06 Histogram Gradient Boosting: The Engineering Inside XGBoost, LightGBM and CatBoost The three dominant boosting libraries share one objective and differ in how they search for splits, grow trees and handle leakage, and those engineering choices decide speed, memory and which datasets each one overfits.
- 07 TreeSHAP and Feature Importance Impurity, permutation and SHAP importance answer three different questions about a tree ensemble, each with a known bias, and TreeSHAP made the game-theoretic one exact and fast without settling which question you should be asking.