Trees & Ensembles
Impurity splitting, bagging, random forests, gradient boosting, and the engineering inside XGBoost and LightGBM.
7concepts
145flashcards
56minutes of reading
- 01 Gradient Boosting as Functional Gradient Descent Boosting is gradient descent in function space, where each new tree approximates the negative gradient of the loss, which is what lets a single algorithm fit any differentiable objective.
- 02 Histogram Gradient Boosting: The Engineering Inside XGBoost, LightGBM and CatBoost The three dominant boosting libraries share one objective and differ in how they search for splits, grow trees and handle leakage, and those engineering choices decide speed, memory and which datasets each one overfits.
- 03 TreeSHAP and Feature Importance Impurity, permutation and SHAP importance answer three different questions about a tree ensemble, each with a known bias, and TreeSHAP made the game-theoretic one exact and fast without settling which question you should be asking.