Trees And Ensembles
4concepts
31flashcards
30minutes of reading
- 01 Decision Trees and Impurity Splitting How a greedy search over axis-aligned splits builds a piecewise-constant function, why Gini and entropy almost never disagree, and the specific structural biases that make a single tree unstable.
- 02 Bagging and Random Forests Why averaging unstable models reduces variance, why bootstrap sampling alone is not enough, and what the extra feature subsampling in a random forest is actually buying.
- 03 Why Trees Still Beat Deep Nets on Tabular Data The three inductive biases that separate tree ensembles from neural networks on tabular problems, and the specific dataset conditions under which the ordering has been observed to flip.
- 04 Gradient Boosting as Functional Gradient Descent Boosting is gradient descent in function space, where each new tree approximates the negative gradient of the loss, which is what lets a single algorithm fit any differentiable objective.