Inside Gradient-Boosted Trees: The Engineering That Made XGBoost, LightGBM and CatBoost Win Tabular ML

Of the 29 prize-winning solutions Kaggle's blog published during 2015, 17 used XGBoost and eight used nothing else. Deep neural networks, the next most common method, appeared in 11 (Chen and Guestrin, 2016, XGBoost, KDD, arXiv:1603.02754). Nine years later the library names had changed but the model family had not: among the 79 winning solutions from 2024 that ML Contests found, 16 used LightGBM, 13 CatBoost and 8 XGBoost (ML Contests, 2024).

The algorithm underneath all three dates from 2001. This post asks what the three libraries changed, and the answer is mechanical. Every split a gradient-boosted tree makes reduces to two running sums per candidate child, the sum of first derivatives \(G\) and of second derivatives \(H\). XGBoost made those sums the objective, LightGBM made them cheap, and CatBoost made them unbiased.

Why this matters: Gradient-boosted trees remain the default for tabular work in fraud, credit, ranking, pricing and forecasting, and their hyperparameters do not transfer across libraries. min_child_weight is a Hessian sum, not a row count. num_leaves is not max_depth. CatBoost's Ordered mode exists because target encoding done the obvious way leaks the label.

TL;DR

• XGBoost's core contribution is a closed-form split score: with a second-order Taylor expansion and an L2 penalty \(\lambda\), a leaf's optimal weight is \(-G/(H+\lambda)\) and a split's gain is half the change in \(G^2/(H+\lambda)\), minus a per-leaf cost \(\gamma\).
• Each row then behaves like a squared-error sample with target \(-g_i/h_i\) and weight \(h_i\), which is why XGBoost's approximate split finder uses a Hessian-weighted quantile sketch.
• A learned default direction for missing values made split search scale with non-missing entries, over 50x faster on a one-hot-heavy Allstate subset.
• On KDD Cup 2012 data LightGBM trained at 12.67 s per iteration against 191.99 s for XGBoost's exact method, with AUC 0.7051 against 0.7029. Feature bundling did most of that; GOSS alone gave nearly 2x.
• CatBoost proved that reusing labels across boosting steps biases the model by a term shrinking like \(1/(n-1)\). Plain boosting's logloss was up to 3.9% worse than Ordered boosting's on small datasets and within about 1% on larger ones; CatBoost's CPU default is Plain.
• Naive target encoding is the bigger leak: greedy target statistics raised logloss by up to 57% over ordered ones.
• No library wins everywhere. Among 19 algorithms on 98 datasets, the best, CatBoost, averaged rank 5.5; on about a third of 176 datasets light tuning mattered more than model family.

At a Glance

flowchart LR
    L["Loss and current predictions"] --> GH["Per-row g and h"]
    GH --> S["Row sampling"]
    S --> B["Bin features into histograms"]
    B --> SC["Scan bins for best gain"]
    SC --> W["Leaf weight -G over H plus lambda"]
    W --> U["Shrink by eta and add tree"]
    U -->|"next round"| L
    X1["XGBoost - score the sums"] -.-> SC
    X2["LightGBM - make sums cheap"] -.-> B
    X3["CatBoost - make sums unbiased"] -.-> GH

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class L,GH blue
    class S,B,SC purple
    class W,U teal
    class X1,X2,X3 amber

The loop is Friedman's; the dashed notes mark where each library spent its effort.

From AdaBoost to a Newton Step in Function Space

AdaBoost combined weak learners by reweighting misclassified examples (Freund and Schapire, 1997, JCSS 55(1)), but what it optimised was unclear. Mason, Baxter, Bartlett and Frean then showed that boosting performs gradient descent on a cost functional (Mason et al., 1999, NIPS 12), and Friedman, Hastie and Tibshirani showed AdaBoost fits an additive logistic model, proposing LogitBoost with Newton steps on the binomial likelihood (Friedman, Hastie and Tibshirani, 2000, Annals of Statistics 28(2)).

Friedman's 2001 paper made it a general recipe (Friedman, 2001, Greedy Function Approximation: A Gradient Boosting Machine, Annals of Statistics 29(5)). For any differentiable loss, compute each row's negative gradient, fit a regression tree to those pseudo-residuals, choose the best constant per leaf, and add the tree scaled by a shrinkage factor. A year later he added per-round row subsampling (Friedman, 2002, CSDA 38(4)).

timeline
    title Gradient-Boosted Trees from Theory to Default
    1997 : Freund and Schapire publish AdaBoost
    1999 : Mason et al. frame boosting as gradient descent
    2000 : Friedman, Hastie and Tibshirani link AdaBoost to additive logistic regression
    2001 : Friedman's gradient boosting machine with shrinkage
    2002 : Stochastic gradient boosting subsamples rows each round
    2011 : pGBRT builds histograms of gradient statistics for ranking
    2016 : XGBoost with second-order split score and sparsity-aware splits
    2017 : LightGBM with GOSS, feature bundling and leaf-wise growth
         : Yandex preprint flags biased gradient estimates
    2018 : CatBoost at NeurIPS with ordered boosting and oblivious trees
    2023 : XGBoost 2.0 makes the histogram method the default
    2025 : TabArena ranks tuned CatBoost first as a single model

What remained was engineering. Classic implementations sorted each feature and tried every threshold; on a million Higgs rows scikit-learn needed 28.51 seconds per tree (Chen and Guestrin, 2016). Gradient histograms already existed in pGBRT for web ranking (Tyree et al., 2011, WWW).

[IMAGE: Two-panel figure. Left: a one-dimensional target with three boosted stumps added in sequence, residuals shrinking each time. Right: the same process drawn as steps on a loss surface in function space, each arrow labelled "tree t fitted to negative gradient". Caption: "Gradient boosting is gradient descent where each step is a tree, not a parameter update."]

How the Three Libraries Actually Work

The objective XGBoost writes down

XGBoost puts the regulariser inside the loss. The model is a sum of trees, \(\hat y_i = \sum_k f_k(x_i)\), trained to minimise

where \(T\) is a tree's leaf count and \(w\) its leaf values (Chen and Guestrin, 2016). \(\gamma\) prices each leaf and \(\lambda\) shrinks leaf values; set both to zero and you recover ordinary gradient tree boosting. At round \(t\) the earlier trees are frozen, and expanding the loss to second order around the current prediction gives

where \(g_i\) and \(h_i\) are the first and second derivatives of the loss with respect to the prediction. For logistic loss on the log-odds scale, \(g_i = p_i - y_i\) and \(h_i = p_i(1-p_i)\). This is the break from Friedman's first-order recipe: curvature enters the search for the tree's structure, not just a leaf-value correction afterwards.

From leaf weights to a split score

A tree maps each row to a leaf \(q(x)\) with weight \(w_j\). Let \(I_j\) be the rows in leaf \(j\). Dropping constants and regrouping by leaf:

Each leaf is an independent quadratic in \(w_j\). Minimising it gives the optimal weight and the best achievable loss for a fixed tree shape:

The weight is a regularised Newton step, gradient over curvature, with \(\lambda\) as a curvature floor. The score depends on the data only through \(G_j\) and \(H_j\). Splitting rows \(I\) into \(I_L\) and \(I_R\) swaps one term for two and adds a leaf, so

That formula is the algorithm. To test every threshold on a sorted feature, sweep once accumulating \(G_L\) and \(H_L\), and get the right side by subtracting from the parent totals. It also explains min_child_weight: a minimum \(H\) per child, which equals a row count for squared error but counts confident logistic rows as tiny fractions of a row (XGBoost parameters).

[IMAGE: A single node's split sweep. A sorted feature column with (g, h) chips per row; a cursor accumulates GL and HL; a line chart above plots gain at each threshold, maximum highlighted. Caption: "Exact split finding is one pass of prefix sums per feature."]

The Hessian as a sample weight

For large data XGBoost proposes a limited set of candidate thresholds and aggregates \(g\) and \(h\) between them. The paper notes that the second-order objective can be rewritten as

which is weighted squared error with target \(-g_i/h_i\) and weight \(h_i\). (The paper prints the target without the minus sign; the weights are unchanged.) Candidate thresholds should therefore split Hessian mass evenly, not row count. XGBoost ranks values by their share of total \(h\) and picks candidates whose adjacent ranks differ by less than \(\epsilon\), about \(1/\epsilon\) of them, using a new mergeable weighted quantile sketch with a provable bound. Confident rows, with tiny \(h_i\), get few boundaries.

Sparsity-aware splits

XGBoost gives each node a default direction for missing values. The scan visits only rows where the feature is present, once with the missing rows' \(G\) and \(H\) assigned right and once assigned left, and keeps the better direction and threshold. Cost scales with non-missing entries, and missingness is learned per node rather than imputed. On Allstate-10K this ran more than 50 times faster than the naive algorithm. Cache-aware and out-of-core block designs then let it train on 1.7 billion Criteo rows with four machines.

LightGBM: attack the data-times-features term

Ke et al. start from a cost model. A histogram learner spends \(O(\text{data} \times \text{features})\) building histograms and only \(O(\text{bins} \times \text{features})\) scanning them, so the way to go faster is fewer rows or fewer features (Ke et al., 2017, LightGBM, NeurIPS). Features are bucketed into at most 255 bins by default, and the histogram subtraction trick builds only the smaller child's histogram, deriving its sibling as parent minus child (LightGBM Features).

Gradient-based One-Side Sampling (GOSS) cuts rows. Rows with small \(|g|\) are already well fitted and contribute little to gain. GOSS keeps the top \(a\) fraction by \(|g|\), samples a \(b\) fraction of the rest, and scales the sampled rows by \((1-a)/b\) so their sum stands in for the discarded population:

\(A\) is the kept set, \(B\) the sample, \(l\) and \(r\) the sides of threshold \(d\) on feature \(j\), and \(n\) terms are row counts: a first-order gain, with counts where XGBoost has Hessian sums. Experiments used \(a = b = 0.05\) or \(0.1\).

Exclusive Feature Bundling (EFB) cuts features. Sparse data holds many features that are almost never non-zero together, one-hot columns being the obvious case, and they can share a histogram if their bin ranges are offset. Optimal bundling is NP-hard, via graph colouring, so LightGBM colours greedily with a tolerance for conflicts. On KDD Cup 2010, with 29 million features, EFB alone cut per-iteration time from 39.85 s to 6.33 s.

Leaf-wise growth changes tree shape. LightGBM always splits the single leaf with the largest loss reduction rather than every node at a depth. At a fixed leaf count this reaches lower training loss and overfits small data more readily, so the capacity control is num_leaves (default 31) (LightGBM Parameters). Categoricals are split by sorting categories on sum_gradient / sum_hessian, about \(O(k \log k)\) instead of \(2^{k-1}-1\) partitions.

CatBoost: the gradient on a row has already seen that row

Prokhorenkova et al. identify a flaw shared by every implementation above (Prokhorenkova et al., 2018, CatBoost: Unbiased Boosting with Categorical Features, NeurIPS, arXiv:1706.09516). The model after \(t-1\) rounds was fitted on every label, so the gradient at training row \(k\) comes from a prediction already pulled toward \(y_k\), and its distribution differs from that on test rows. They call this prediction shift and size it in a toy case with two Bernoulli features, \(y = c_1 x^1 + c_2 x^2\), and two boosting steps with stumps. Independent samples per step give an unbiased model; reusing one sample of size \(n\) gives

The same leak is far worse in target statistics. Greedy encoding replaces a category with a smoothed mean of \(y\) over its rows, including the current row. If every category is unique and \(P(y=1) = 0.5\), one split then classifies the training set perfectly while test accuracy is 0.5. Leave-one-out encoding still leaks: on a constant feature it separates the classes perfectly.

CatBoost's fix is an artificial arrow of time. Draw a random permutation. Compute each row's target statistic only from rows before it (ordered TS), and compute its residual from a model trained only on rows before it (ordered boosting). Keeping one model per prefix is infeasible, so CatBoost stores supporting predictions for prefixes of length \(2^j\), cutting storage from \(O(sn^2)\) to \(O(sn)\) for \(s\) permutations, and uses several permutations because rows early in a permutation have high-variance estimates. Ordered mode ran about 1.7 times slower than Plain on Epsilon.

The base learner is an oblivious tree, where every node at a given depth uses the same split. The leaf index is the \(d\)-bit integer formed by \(d\) tests, so scoring is a table lookup. The engineering paper reports scoring an 8,000-tree Epsilon model in 2.4 s on one thread, against 78 s for XGBoost and 122 s for LightGBM (Dorogush, Ershov and Gulin, 2018, arXiv:1810.11363).

[IMAGE: Three tree shapes at a budget of 8 leaves. Left: level-wise depth-3 tree with different features per node. Middle: leaf-wise tree with one branch reaching depth 6. Right: oblivious tree repeating one test per level, leaves labelled by 3-bit codes 000 to 111. Caption: "Same leaf count, three shapes: level-wise balances, leaf-wise chases loss, oblivious trades flexibility for regularisation and lookup-table inference."]

Seeing It in Motion

Where each library intervenes in one round:

flowchart TB
    subgraph XGB["XGBoost 2016"]
        XA["Second-order g and h"] --> XB["Weighted quantile sketch"]
        XB --> XC["Sparsity-aware scan"]
        XC --> XD["Gain minus gamma, depth-wise"]
    end
    subgraph LGB["LightGBM 2017"]
        LA["GOSS keeps top a, samples b"] --> LB["EFB bundles exclusive features"]
        LB --> LC["Histograms plus subtraction"]
        LC --> LD["Leaf-wise growth"]
    end
    subgraph CB["CatBoost 2018"]
        CA["Random permutations"] --> CBB["Ordered target statistics"]
        CBB --> CC["Ordered residuals"]
        CC --> CD["Oblivious tree"]
    end
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    class XA,LA,CA blue
    class XB,XC,LB,LC,CBB,CC purple
    class XD,LD,CD teal

The expensive step is the histogram build, done for one child per split.

sequenceDiagram
    participant B as Booster
    participant O as Objective
    participant S as Sampler
    participant H as Histogram builder
    participant F as Split finder
    B->>O: Current raw scores
    O-->>B: g and h per row
    B->>S: Rows with g and h
    S-->>H: Top-gradient rows plus reweighted sample
    H->>F: Parent histograms of G and H
    F->>F: Scan bins and pick best leaf
    F->>H: Build smaller child only
    H-->>F: Larger child equals parent minus smaller
    Note over F,H: Repeat until leaf limit or no gain beats gamma
    F-->>B: Tree with leaf weights
    B->>B: Add eta times tree to scores

By the Numbers

Speed figures come from each library's authors on 2016 to 2018 versions: evidence for mechanisms, not current benchmarks.

Sources: Chen and Guestrin, 2016; Ke et al., 2017; Prokhorenkova et al., 2018; Dorogush et al., 2018; ML Contests, 2024; McElfresh et al., 2023. CatBoost's comparison was run by its authors, who encoded categoricals for all three libraries with ordered target statistics. ML Contests' counts cover the winning solutions it could review, not every competition.

[IMAGE: Log-scale bar chart of LightGBM paper per-iteration times on KDD10 and KDD12, bars for xgb exact, lgb baseline, EFB only and full LightGBM, with the EFB step annotated as the largest drop. Caption: "On very wide sparse data, bundling features buys far more than sampling rows."]

A Concrete Example

One split decision by hand, the way XGBoost's exact method makes it. Churn prediction with logistic loss; one node holds seven customers with one feature, \(x\) = days since last login, missing for one customer. A previous tree has produced probabilities \(p\). Use \(\lambda = 1\), \(\gamma = 0\), \(\eta = 0.3\).

Step 1: gradients and Hessians, \(g = p - y\) and \(h = p(1-p)\).

Step 2: parent score. \(G = -0.4\), \(H = 1.40\), so \(G^2/(H+\lambda) = 0.16/2.40 = 0.0667\).

Step 3: threshold 7, missing sent right. Left holds rows 1 and 2: \(G_L = 0.6\), \(H_L = 0.40\), term \(0.36/1.40 = 0.2571\). Right holds rows 3 to 7: \(G_R = -1.0\), \(H_R = 1.00\), term \(1.00/2.00 = 0.5000\). Gain \(= \tfrac{1}{2}(0.2571 + 0.5000 - 0.0667) = 0.3452\).

Step 4: same threshold, missing sent left. Now \(G_L = 0.5\), \(H_L = 0.49\), term \(0.1678\); \(G_R = -0.9\), \(H_R = 0.91\), term \(0.4241\). Gain \(= \tfrac{1}{2}(0.1678 + 0.4241 - 0.0667) = 0.2626\).

Step 5: the full scan.

The winner is \(x < 7\) with missing values defaulting right, where customer 7's gradient fits. The negative gain at 11.5 is \(\lambda\) at work: a split that barely separates gradients scores below none.

Step 6: leaf weights and update. \(w_L = -0.6/1.4 = -0.429\) and \(w_R = 1.0/2.0 = +0.500\). After shrinkage the log-odds move by \(-0.129\) and \(+0.150\). Row 3 goes from \(\ln(0.3/0.7) = -0.847\) to \(-0.697\), probability 0.300 to 0.332.

Step 7: what it bought. Mean logloss falls from 0.5149 to 0.4845, though row 6 worsens from 0.693 to 0.771, left for the next tree. With \(\gamma = 0.35\) the node would stay unsplit.

The Hessian mattered. Scoring the six non-missing rows with the first-order, count-based variance gain (relative to the parent), thresholds 7 and 25.5 nearly tie at 0.0613 and 0.0605, while the second-order score separates them, 0.3452 against 0.2550. And approximation keeps it: three buckets of roughly equal Hessian mass put candidates at 7 and 17.

[IMAGE: Seven customer chips on a number line of days since login, coloured by label and annotated with (g, h), the missing-value chip floating above; beneath, gain curves for both default directions with 7.0/right starred. Caption: "One split from two running sums: G and H choose the threshold, the missing-value direction and the leaf values."]

Where It Breaks

Hessians that vanish or do not exist

The Newton step \(-G/(H+\lambda)\) assumes informative positive curvature. For logistic loss \(h = p(1-p)\) collapses as predictions grow confident, so leaves of confident rows can take large weights unless \(\lambda\) or a cap intervenes; XGBoost's documentation says max_delta_step "might help in logistic regression when class is extremely imbalanced". Conversely min_child_weight = 1 can block splitting hundreds of confident rows whose Hessians sum below one. For absolute or quantile loss the second derivative is zero almost everywhere, the derivation does not apply as written, and libraries fall back on workarounds.

Leaf-wise growth on small data

On a few thousand rows leaf-wise growth finds deep, narrow branches that fit noise, and trees still grow leaf-wise when max_depth is set. Setting 64 leaves to "match" XGBoost's depth 6 permits far deeper trees.

Binning throws away resolution

Histograms quantise each feature to about 255 bins (254 borders by default in CatBoost on CPU) (CatBoost quantization). A threshold effect in a narrow range of a wide feature may fall between borders. scikit-learn's guide says its exact estimator "might be preferred for small sample sizes since binning may lead to split points that are too approximate" (scikit-learn ensembles).

Sampling and bundling are approximations

GOSS's \((1-a)/b\) amplification is noisy for small \(b\) on small data, and it saves less than its sampling rate because gradients are still computed on every row: 10 to 20 percent of rows gave about 2x. EFB with conflict tolerance blurs features that sometimes co-occur.

Target encoding leaks, whoever does it

Ordered statistics protect you only inside CatBoost. Target-encoding categoricals on the full training set before any GBDT recreates the greedy case.

Oblivious trees restrict each tree

A depth-6 oblivious tree has 64 leaves but only 6 tests, so asymmetric interactions need more trees. CatBoost now also offers Depthwise and Lossguide growth (CatBoost parameters).

Alternative Designs

The libraries have converged. XGBoost 2.0 (September 2023) made hist the default (XGBoost 2.0.0 release notes); it offers leaf-wise growth (lossguide) and a gradient_based sampler proportional to \(\sqrt{g^2 + \lambda h^2}\), and scikit-learn's HistGradientBoosting is described as inspired by LightGBM. What still differs is categorical handling, tree shape and defaults.

How It Is Used in Practice

Which library wins is contested

Each paper reports its own library ahead. CatBoost beat both rivals on all nine test datasets, significantly on six, but its authors ran the baselines. Independent work is less tidy. Anghel et al. compared GPU versions under Bayesian hyperparameter search and found "no clear winner in terms of time-to-solution": XGBoost gained most from GPUs, LightGBM sometimes generalised better, and CatBoost converged fastest on very wide data where XGBoost ran out of memory (Anghel et al., 2018, arXiv:1809.04559). McElfresh et al. found nearly every algorithm ranked first on at least one dataset and last on another (McElfresh et al., 2023, NeurIPS Datasets and Benchmarks, arXiv:2305.02997). TabArena, over 51 curated datasets, ranked CatBoost first as a single tuned model but placed LightGBM in the top three once configurations were ensembled (Erickson et al., 2025, TabArena, NeurIPS Datasets and Benchmarks, arXiv:2506.16791). The defensible reading: library choice matters less than tuning, validation and features.

Operating them

Latency-sensitive scoring favours CatBoost's oblivious trees; wide sparse training favours LightGBM. GPU and distributed training are XGBoost strengths; its GPU tree builder is described by Mitchell and Frank, 2017, PeerJ Computer Science 3:e127. With any of them: early-stop on a time-respecting validation split, version bin borders and category mappings with the model, and target-encode only inside the library or inside folds.

[IMAGE: Decision flowchart: "many high-cardinality categoricals?" leads to CatBoost; "millions of sparse columns?" to LightGBM; "GPU cluster or out-of-core data?" to XGBoost hist; every path ends at "tune, early-stop, consider cross-library ensembling". Caption: "Route by data shape and operating constraints, not by a universal winner."]

Insights Worth Remembering

1. A gradient-boosted tree is a machine for computing \(G\) and \(H\). Thresholds, leaf values, pruning, missing-value routing and min_child_weight all come from two sums. When a model misbehaves, inspect those sums in the offending leaves.

2. Second-order boosting is weighted least squares each round. Targets \(-g/h\) with weights \(h\) explain the weighted sketch, the leaf formula, and why confident rows barely move the next tree.

3. LightGBM's biggest measured win was cutting features, not rows. On its widest datasets EFB cut per-iteration time 6 to 8 times; GOSS added about 2x. Width was the bottleneck.

4. Prediction shift is real and usually small. The bias scales like \(1/(n-1)\), and Ordered mode helped most on Adult and Internet, two datasets under 40,000 training rows; on seven others Plain was within 0.6%. CatBoost's own CPU default is Plain.

5. Categorical leakage is the common, expensive error. Greedy target statistics cost tens of percent in logloss, and the lesson applies to every hand-built encoding pipeline.

6. Benchmarks crown CatBoost, competitions crown LightGBM, and both can be true. Lightly tuned comparisons reward strong defaults; unlimited-effort competitions reward iteration speed and ensembling.

Open Questions

How much of CatBoost's benchmark strength comes from ordering? The original ablations show ordering matters on small data. What is not established by published independent ablations is how much of CatBoost's average rank on large benchmarks comes from symmetric trees and defaults instead.

Is second-order information worth it for every loss? Newton steps help where curvature varies, as in logistic loss; systematic first- versus second-order comparisons at matched tuning are scarce.

Can GOSS and EFB be tuned from data properties? Their guarantees are theoretical, and the LightGBM authors listed choosing \(a\) and \(b\) as future work.

Will ensembled deep models or foundation models displace GBDTs as defaults? TabArena measured ensembled neural networks at the top of its leaderboard and TabPFNv2 leading within its size limits. Whether that holds at millions of rows, where trees are cheapest, and at acceptable inference cost, is unresolved.

Sources and Further Reading

1. Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine." The Annals of Statistics, 29(5), 1189-1232. doi:10.1214/aos/1013203451
2. Chen, T., & Guestrin, C. (2016). "XGBoost: A Scalable Tree Boosting System." KDD 2016, 785-794. arXiv:1603.02754
3. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). "LightGBM: A Highly Efficient Gradient Boosting Decision Tree." NeurIPS 2017. Proceedings
4. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). "CatBoost: Unbiased Boosting with Categorical Features." NeurIPS 2018. arXiv:1706.09516
5. Dorogush, A. V., Ershov, V., & Gulin, A. (2018). "CatBoost: Gradient Boosting with Categorical Features Support." arXiv:1810.11363
6. Friedman, J. H. (2002). "Stochastic Gradient Boosting." Computational Statistics and Data Analysis, 38(4), 367-378. doi:10.1016/S0167-9473(01)00065-2
7. Friedman, J., Hastie, T., & Tibshirani, R. (2000). "Additive Logistic Regression: A Statistical View of Boosting." The Annals of Statistics, 28(2), 337-407. doi:10.1214/aos/1016218223
8. Mason, L., Baxter, J., Bartlett, P., & Frean, M. (1999). "Boosting Algorithms as Gradient Descent." NIPS 12. Proceedings
9. Freund, Y., & Schapire, R. E. (1997). "A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting." JCSS, 55(1), 119-139. doi:10.1006/jcss.1997.1504
10. Tyree, S., Weinberger, K., Agrawal, K., & Paykin, J. (2011). "Parallel Boosted Regression Trees for Web Search Ranking." WWW 2011, 387-396.
11. Mitchell, R., & Frank, E. (2017). "Accelerating the XGBoost Algorithm Using GPU Computing." PeerJ Computer Science, 3, e127. doi:10.7717/peerj-cs.127
12. Anghel, A., Papandreou, N., Parnell, T., De Palma, A., & Pozidis, H. (2018). "Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms." arXiv:1809.04559
13. McElfresh, D., et al. (2023). "When Do Neural Nets Outperform Boosted Trees on Tabular Data?" NeurIPS 2023 Datasets and Benchmarks. arXiv:2305.02997
14. Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., Desai, P. M., Salinas, D., & Hutter, F. (2025). "TabArena: A Living Benchmark for Machine Learning on Tabular Data." NeurIPS 2025 Datasets and Benchmarks. arXiv:2506.16791
15. ML Contests. (2024). "The State of Machine Learning Competitions." mlcontests.com
16. Library documentation: XGBoost 2.0.0 release, XGBoost parameters, LightGBM features, LightGBM parameters, CatBoost parameters, scikit-learn ensembles

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.