The Last Undefeated Baseline: Why Gradient-Boosted Trees Still Beat Deep Learning on Tabular Data

In 2022, three researchers at Inria ran what remains the most careful comparison in this area: 45 curated tabular datasets, a standardised benchmarking protocol, and a hyperparameter search budget of 20,000 compute hours per learner. Their conclusion was that tree-based models remain state of the art on medium-sized tabular data of around 10,000 samples, before accounting for their enormous speed advantage (Grinsztajn, Oyallon and Varoquaux, 2022, NeurIPS Datasets and Benchmarks, arXiv:2207.08815).

That is a strange result. The same architecture family that solved object recognition, machine translation, protein folding and code generation cannot reliably beat a technique from 2001 on the most common data format in commercial computing. It is not for lack of trying: TabNet, NODE, TabTransformer, SAINT, FT-Transformer, DeepFM and dozens more have each claimed to close the gap. What makes the result interesting is that the Inria paper did not stop at the leaderboard. It went looking for why, and the answer turns out to be three specific, testable properties of the data rather than anything about optimisation or scale.

Why this matters: Most production machine learning is tabular. If you are choosing between a gradient-boosted tree and a neural network for a churn model, a credit decision, a fraud score or a demand forecast, the literature has a clear default and a set of conditions under which the default flips. Knowing those conditions saves a quarter of engineering time per project, and knowing why they exist tells you which new architecture claims are worth reading.

TL;DR

• On 45 curated benchmarks with a 20,000-compute-hour tuning budget per learner, tree ensembles remained ahead of every tested deep architecture at around 10,000 samples, before counting their large speed advantage.
• The gap has three named causes, each established by an intervention rather than asserted: neural networks are biased toward smooth functions, they are hurt disproportionately by uninformative features, and their rotation invariance discards the meaning of the column basis.
• Rotation invariance is a feature on images, where the pixel basis is arbitrary, and a liability on tables, where "age" and "income" are not an arbitrary basis.
• An independent 2021 study found XGBoost beat every proposed deep model on the datasets those models' own papers used, while needing far less tuning; an ensemble of both beat XGBoost alone.
• A second 2021 study, which built stronger deep baselines than most papers bother with, concluded the opposite of a clean win in either direction: "there is still no universally superior solution."
• The ordering does flip. TabPFN, a transformer pre-trained on synthetic datasets that performs in-context Bayesian inference with no gradient steps at fit time, reported outperforming prior methods on datasets up to about 10,000 samples and reached Nature in 2025.
• Under equal wall-clock budgets rather than equal trial counts, the tree advantage widens, because a laptop CPU can run a hundred XGBoost configurations in the time one GPU tunes an MLP.
• The comparison stops being the relevant one the moment the problem includes text, images, a related large corpus to pretrain on, or hundreds of millions of rows with high-cardinality categoricals.

At a Glance

flowchart LR
    D["Tabular dataset<br/>rows x named columns"] --> P{"What does the<br/>data look like?"}
    P -->|"~10k rows, many<br/>weak features,<br/>irregular targets"| T["Tree ensemble<br/>XGBoost / LightGBM"]
    P -->|"100M+ rows,<br/>high-cardinality IDs"| E["Embedding network<br/>learned representations"]
    P -->|"text or image<br/>columns present"| N["Neural, shared<br/>representation"]
    P -->|"under ~10k rows,<br/>numeric features"| F["Prior-fitted network<br/>TabPFN"]
    T --> R["Prediction"]
    E --> R
    N --> R
    F --> R

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class D blue
    class P slate
    class T emerald
    class E,N,F purple
    class R teal

The decision is not "which model is better" but "which prior matches this data", and the four branches correspond to genuinely different regimes rather than to preference.

How Tabular Data Became Deep Learning's Holdout

The pattern in every other domain was the same. A hand-engineered feature pipeline dominated, a neural network learned the features instead, and within a few years nobody used the pipeline. SIFT and HOG gave way to convolutional features. Bag-of-words and parse trees gave way to contextual embeddings. Mel-frequency cepstral coefficients gave way to learned audio front ends.

Tabular data never had that pipeline to replace. The columns are the features, and they arrived already meaningful: someone decided that "days since last login" was worth recording. There is no representation-learning problem to solve, which removes the mechanism that made deep learning transformative everywhere else.

What tabular data had instead was a steadily improving tree lineage. Breiman's random forests in 2001 established that averaging decorrelated high-variance models works. Friedman's gradient boosting, in the same year, reframed boosting as gradient descent in function space, which meant any differentiable loss could be optimised by fitting weak learners to its negative gradient (Friedman, 2001, Annals of Statistics 29(5)). Everything after that was engineering, and the engineering was extraordinarily effective.

timeline
    title Two Lineages, One Benchmark
    1984 : Breiman et al. publish CART
         : Greedy recursive partitioning on axis-aligned splits
    2001 : Random Forests formalise bagging plus feature subsampling
         : Friedman frames boosting as functional gradient descent
    2016 : XGBoost adds second-order objectives and explicit regularisation
         : Becomes the default winner of tabular competitions
    2017 : LightGBM adds histogram binning and leaf-wise growth; over 20x speedup
         : First serious wave of tabular neural architectures begins
    2021 : Shwartz-Ziv and Armon find XGBoost beats deep models on the deep papers' own datasets
         : Gorishniy et al. build stronger deep baselines and find no universal winner
    2022 : Grinsztajn et al. benchmark 45 datasets at 20,000 compute hours per learner
         : Name three inductive biases that explain the gap
    2025 : TabPFN reaches Nature, winning below roughly 10,000 samples via in-context inference

[IMAGE: Two-lane timeline graphic. Upper lane, "trees", shows CART 1984, RF and GBM 2001, XGBoost 2016, LightGBM 2017, CatBoost 2018, each with a small icon of a branching structure. Lower lane, "neural", shows the tabular architecture wave from 2017 onward with dates for TabNet, NODE, TabTransformer, FT-Transformer, SAINT, TabPFN. Vertical dashed lines mark the three benchmark studies of 2021-2022 that evaluated both lanes together. Caption: "The tree lineage had a fifteen-year head start and never stopped improving."]

[IMAGE: Split diagram contrasting "representation learning had something to replace" against "it did not". Left panel: a photograph reduced to a SIFT keypoint diagram, with an arrow to a learned convolutional feature map, labelled "hand-engineered pipeline replaced". Right panel: a spreadsheet with named column headers, with a crossed-out arrow to an empty box, labelled "the columns are already the features". Caption: "Deep learning replaced feature engineering. Tabular data never had the feature engineering step to replace."]

By 2020 the field had a large and unreliable literature: dozens of tabular neural architectures, each reporting a win on a benchmark of its own choosing, with no shared protocol. Two papers in 2021 and one in 2022 changed that, and it is worth reading them as a set, because they do not agree.

What the Three Studies Actually Found

Shwartz-Ziv and Armon: the adversarial replication

The first move was the obvious one that nobody had made. Take the deep models that claimed to beat XGBoost, and evaluate them on the datasets from their own papers, under a shared tuning protocol.

The result was unambiguous: XGBoost outperformed the deep models across the datasets, including the datasets used in the papers proposing those models, and it required far less tuning to get there (Shwartz-Ziv and Armon, 2022, Tabular Data: Deep Learning is Not All You Need, Information Fusion, arXiv:2106.03253). The paper is a study in how much of a reported improvement can come from asymmetric tuning effort: an author tunes their own architecture carefully and runs the baseline with defaults, and the resulting number is a fact about attention allocation rather than about the model.

One finding cuts the other way and is usually left out of the summary: an ensemble of the deep models and XGBoost beat XGBoost alone. The deep models were learning something complementary, even while losing individually.

Gorishniy et al.: raising the deep baselines

The second paper attacked the same problem from the opposite side. If the tabular deep learning literature suffers from weak baselines, build strong ones. Gorishniy and colleagues identified two simple architectures that most papers had not bothered to try properly: a ResNet-style network, and a straightforward adaptation of the Transformer to tabular features, which they called FT-Transformer (Gorishniy et al., 2021, Revisiting Deep Learning Models for Tabular Data, NeurIPS, arXiv:2106.11959).

The ResNet baseline turned out to be strong enough to beat much of the specialised architecture literature, and FT-Transformer outperformed the other deep solutions on most tasks. Their conclusion about the headline question is more careful than either camp usually quotes: comparing the best deep models against GBDT, "there is still no universally superior solution."

Reading the two together, the honest 2021 position was: the deep tabular literature was overstating its wins, GBDT is the right default, and a properly built deep model is competitive on some problems.

Grinsztajn et al.: finding the mechanism

The third paper is the one that made the question scientific rather than empirical. Having established that trees lead on a 45-dataset benchmark, the authors ran interventions on the data itself to isolate which property trees were exploiting.

Intervention one: smooth the target. Apply a Gaussian kernel smoother to the target function of a real dataset and re-evaluate. Tree performance degrades markedly; neural network performance barely moves. This localises one advantage precisely: trees are better because the target is irregular, and removing the irregularity removes their edge. Tabular targets are full of genuine discontinuities, a credit limit, an age boundary, a plan tier threshold, and a tree expresses a step with one split while an MLP must approximate it with many units against an optimiser that prefers not to.

Intervention two: remove uninformative features. Delete features in increasing order of importance and re-evaluate. Much of the gap closes. Add uninformative features and it widens. A tree's greedy split search simply never selects a useless column, so no capacity is consumed by it; a dense layer assigns every input a weight and must learn to zero it, which costs both samples and effective capacity. Real tabular datasets are full of near-useless columns.

Intervention three: rotate the feature space. Apply a random rotation and re-evaluate. MLP performance is essentially unchanged; tree performance collapses. This confirms that MLPs are rotationally invariant in a strong sense and trees are not, and it identifies non-invariance as an advantage, because in tabular data the given basis is meaningful. "Age" and "income" are not an arbitrary coordinate system, and a method that treats them as one has thrown away information that was handed to it for free.

That third result is the conceptual core. On images the pixel basis genuinely is arbitrary, so architectures that impose spatial equivariance and are otherwise indifferent to the basis encode the right prior. On tables, the basis carries the domain knowledge of whoever designed the schema.

graph TD
    subgraph Tree["Tree ensemble prior"]
        T1["Axis-aligned splits"] --> T2["Basis is meaningful:<br/>non-rotation-invariant"]
        T1 --> T3["Unused columns cost nothing"]
        T1 --> T4["One split = one exact threshold"]
    end
    subgraph Net["Neural network prior"]
        N1["Dense linear layers"] --> N2["Rotation invariant:<br/>basis is arbitrary"]
        N1 --> N3["Every input gets a weight<br/>to learn away"]
        N1 --> N4["Smooth functions preferred<br/>by SGD and weight decay"]
    end
    T2 --> W["Wins when columns<br/>carry meaning"]
    T3 --> W
    T4 --> W
    N2 --> L["Wins when a shared<br/>representation must be learned"]
    N3 --> L
    N4 --> L

    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff

    class T1,T2,T3,T4 emerald
    class N1,N2,N3,N4 purple
    class W,L teal

[IMAGE: Three side-by-side scatter plots on the same 2D feature space with a target coloured red/blue. Panel (a) shows an irregular step-like decision boundary with a tree's staircase approximation overlaid, fitting closely. Panel (b) shows the same boundary Gaussian-smoothed, with an MLP's smooth boundary now fitting equally well. Panel © shows the original boundary after a 30-degree rotation, with the tree's staircase now requiring many more steps to approximate it. Caption: "Three interventions, three isolated causes: irregularity, uninformative features, and orientation."]

Why the Mechanism Explains the Engineering

The three inductive biases account for the benchmark results, and they also explain a set of practical advantages that rarely make it into an accuracy table.

No preprocessing. Tree splits depend only on the ordering of a feature's values, so any strictly monotone transformation, log, square root, standardisation, produces an identical tree. No scaling, no normalisation, no skew correction, no distribution assumptions. Every one of those steps is a fitted transform that a neural pipeline needs, must not leak across the train-test split, and can drift in production.

Mixed types without design work. High-cardinality categoricals need an encoding decision in a neural pipeline: embedding dimension, initialisation, handling of unseen levels. Trees consume them through target encoding or native categorical support with a single hyperparameter.

Missing values without a policy. XGBoost learns a default direction per split, sending missing values to whichever side reduces the loss, so missingness becomes usable information rather than a value to impute. A neural network needs an explicit imputation strategy, and that strategy is another fitted artefact.

Speed as a search multiplier. This is the advantage most under-weighted in benchmark papers, because it is invisible when every method gets the same number of trials. A gradient-boosted model on 10,000 rows trains in seconds on a laptop CPU. Under an equal wall-clock budget, the tree runs a hundred configurations while the network runs five, so the tree's realised performance is drawn from a much better-explored hyperparameter space. The Inria paper's own framing acknowledges this by noting the tree result holds "even without accounting for their superior speed".

[IMAGE: Two horizontal budget bars of equal length labelled "two hours of wall clock". The upper bar, "XGBoost", is subdivided into roughly 1,000 thin tick marks, each a completed configuration. The lower bar, "tuned MLP", is subdivided into about 80 wide blocks. Below each, a small scatter of trial scores showing the tree's search densely covering its space and the network's sparsely sampling a larger one. Caption: "Equal-trial benchmarks hide the budget that actually binds a project."]

By the Numbers

Sources: Grinsztajn et al., 2022; Shwartz-Ziv and Armon, 2022; Gorishniy et al., 2021; Ke et al., 2017; Hollmann et al., 2025, Nature 637. The "typical training hardware" and "preprocessing" rows are characterisations of standard practice, not measured quantities. Benchmark results are aggregates across dataset collections and do not transfer to any individual dataset.

A Concrete Example

Take a small, realistic problem and trace what each model does with it. A lender predicts default from four features, on 8,000 historical applications with a 6% default rate.

Step 1: the threshold. The real target has a discontinuity: default probability is 11% just below income = 45000 and 7% just above, because applications below the line skip a review step. A single tree split at 45,000 captures this exactly and costs one node. An MLP must build the step from a sum of sigmoids; with weight decay and a few thousand samples it produces a ramp spanning roughly 40k to 50k, systematically under-predicting risk at 44,000 and over-predicting at 46,000. This is intervention one, visible in one feature.

Step 2: the useless column. application_hour carries no signal. The tree's split search evaluates it at every node, never finds an impurity reduction that beats the alternatives, and never uses it; the model behaves as though the column were absent. The MLP's first layer assigns it 64 weights (one per hidden unit), all of which must be driven toward zero by the optimiser. On 8,000 rows those weights instead settle on small non-zero values fitting noise, and the model has spent a twenty-fifth of its first-layer capacity learning nothing. This is intervention two.

Step 3: the rotation. Suppose you standardise and apply PCA before the network, a common preprocessing step. The first component is roughly \(0.7\,\widehat{\texttt{income}} + 0.7\,\widehat{\texttt{months}}\). The 45,000 threshold, which was a clean cut on one axis, is now a diagonal boundary in the rotated space, and expressing it requires the tree to build a staircase. The MLP is unaffected. You have just handed the neural network a fair fight by throwing away the information that income was a distinguished direction. This is intervention three, and it is why "PCA before the model" is a worse default on tabular data than it is on signals.

Step 4: the count. Fit both under a two-hour wall-clock budget on a laptop. XGBoost, at roughly 4 seconds per fit with 300 trees on 8,000 rows, completes on the order of a thousand configurations of max_depth, learning_rate, min_child_weight, subsample and colsample_bytree. The MLP, at roughly 90 seconds per fit including early stopping, completes about 80 configurations of width, depth, learning rate, dropout, weight decay and embedding dimension, over a search space that is larger. The tree is being evaluated near its ceiling and the network is not.

Step 5: the numbers. Suppose the outcome is AUC 0.782 for XGBoost and 0.771 for the tuned MLP. That 0.011 gap is not evidence that neural networks cannot model this problem. It decomposes into a genuine inductive-bias advantage on steps 1 and 2, an artefact of preprocessing in step 3 that you introduced, and a search-budget asymmetry in step 4. Only the first is a property of the models.

[IMAGE: Line chart with income on the x-axis from 30k to 60k and predicted default probability on the y-axis. Three lines: the true step function dropping sharply at 45k; the tree ensemble's near-exact step; the MLP's smooth sigmoid ramp spanning 40k-50k. Shaded regions show the systematic error band on either side of the threshold. Caption: "One policy threshold, one split for a tree, and a systematic error band for a smooth model."]

Where It Breaks

"Tabular" is not one regime

The tree advantage is documented most clearly at thousands to tens of thousands of rows. At hundreds of millions of rows with high-cardinality categorical identifiers, the picture inverts completely: industrial advertising and recommendation systems are dominated by embedding-based networks, not by GBDT, and have been for years. Learning a dense representation for each of ten million user IDs is something a tree cannot do at all, and it is the whole game in that domain.

The regime boundary is not sharp, and it is set by more than row count: cardinality, the presence of ID-like features, and whether representations need to be shared across tasks all move it.

Trees cannot extrapolate

A tree predicts a constant within each leaf, so beyond the range of the training data it returns the nearest leaf's value, forever. For any target with a trend, a growing metric, an inflating price, a series with drift, this is fatal, and no amount of ensembling repairs it: an average of models that cannot extrapolate also cannot extrapolate.

The standard fix lives outside the model. Difference the series, or fit the trend explicitly and give the tree only the detrended residual. Forecasting pipelines that hand raw levels to a gradient-boosted model and are surprised by flat forecasts are hitting a structural limit, not a tuning problem.

No transfer, no pretraining, no multimodality

A tree ensemble learns one dataset from scratch every time. It cannot be pretrained on a related corpus, cannot share a representation with an image or text column, and cannot be fine-tuned. Where any of those apply, and they increasingly do, the comparison is not close, because the neural model is doing something the tree has no mechanism for.

This is the most likely route by which the ordering eventually changes for good: not a better tabular architecture, but tabular data stopping being purely tabular.

Benchmark aggregates say nothing about your dataset

A result across 45 datasets is a statement about a distribution of problems. Your problem is one draw from a different distribution. The variance across datasets in these studies is large, and the cost of settling it directly, one tuned XGBoost against one tuned neural model, is measured in hours.

Benchmark curation is itself contested. Which datasets are admitted, how categoricals are encoded, whether the tuning budget is counted in trials or in hours, and how missing values are handled all move the aggregate ordering. A reader should treat these papers as evidence about typical behaviour, not as a settled ranking.

Asymmetric tuning is the field's recurring methodological failure

The Shwartz-Ziv and Armon result deserves more weight than it usually gets, because it is a claim about how the literature is produced rather than about models. An author tunes their architecture over hundreds of configurations and runs XGBoost with defaults. The reported gap is real in the paper and vanishes under equal effort. Anyone reading a new tabular architecture claim should look first for the baseline's tuning protocol, and treat its absence as disqualifying.

Alternative Designs

The last row is the most under-used option. Shwartz-Ziv and Armon found the ensemble beat XGBoost alone even though every individual deep model lost to it, which means the two families make partly uncorrelated errors. That is exactly the condition under which ensembling pays, and it is a stronger practical result than either half of the horse race.

[IMAGE: Grouped bar chart with dataset size on the x-axis in log scale, from 1,000 to 100,000,000 rows, and relative performance on the y-axis. Four bands: TabPFN leading below 10,000; GBDT leading from 10,000 to roughly 10 million; embedding networks leading above that; a shaded overlap zone between each pair marked "depends on cardinality and feature types". Caption: "The regime boundaries are approximate and set by more than row count."]

How It Is Used in Practice

The pattern in production is more settled than the literature suggests, because production has a cost function the papers do not.

Credit and risk. Gradient-boosted trees dominate, and the reason is only partly accuracy. Regulatory environments require explanation of individual decisions, and while SHAP values are computable for any model, tree SHAP has an exact polynomial-time algorithm where the model-agnostic version is a sampling approximation. Monotonic constraints, forcing the model to be non-increasing in income, are a first-class feature in XGBoost and LightGBM and are not straightforward in a network. A regulator asking why an application was declined gets a better answer from a tree.

Fraud and abuse. Trees again, for a different reason: the label distribution and the adversary both move constantly, so retraining frequency dominates model sophistication. A model retrainable in minutes on commodity hardware beats a marginally better one that takes a day.

Advertising and recommendation. Embedding networks, decisively. The features are dominated by high-cardinality identifiers, the data is measured in billions of rows, and representations are shared across many prediction tasks. This is the regime where the tree's inability to learn representations is disqualifying rather than merely limiting.

Forecasting. Gradient-boosted trees on lag features are extremely competitive, and every serious pipeline handles the extrapolation problem explicitly by differencing or detrending before the model sees the data. The M-competition results made cross-learning across series standard, and both global GBDT and global neural forecasters exploit it.

The operational tail. A tree ensemble is a set of thresholds. It serves on CPU, has no GPU dependency, has predictable latency, serialises to a few megabytes, and can be scored in a database or a rules engine. Anyone who has carried a GPU-serving requirement into an environment that did not want one understands that this is not a minor consideration.

stateDiagram-v2
    [*] --> Baseline
    Baseline: Tuned GBDT, one afternoon
    Baseline --> Sufficient: metric meets the bar
    Baseline --> Investigate: metric falls short
    Investigate --> Features: add or fix features
    Features --> Baseline
    Investigate --> Regime: check the regime
    Regime --> Embeddings: 100M rows or ID-heavy
    Regime --> Neural: text or image columns present
    Regime --> Small: under 10k rows
    Small --> PFN: try a prior-fitted network
    Embeddings --> Ensemble
    Neural --> Ensemble
    PFN --> Ensemble
    Ensemble: Consider GBDT + neural ensemble
    Ensemble --> Sufficient
    Sufficient --> [*]

The ordering in that diagram is the practical claim of this whole article: start with the tree, because it is an afternoon's work and a strong result, and move only when a named condition tells you to.

[IMAGE: Annotated screenshot-style mockup of a SHAP force plot for a single credit decision from a gradient-boosted model, showing the base rate, four feature contributions with magnitudes, and the final prediction. Annotation arrows point to the exactness of the tree SHAP computation versus the sampling approximation used for model-agnostic SHAP. Caption: "Exact per-feature attribution is a property of the model class, not an add-on."]

Insights Worth Remembering

1. Rotation invariance is a prior, and priors are only good when they match. Being indifferent to the coordinate system is exactly right for pixels and exactly wrong for named columns. The property that made MLPs general-purpose is the property that costs them here, which is a cleaner statement of the tradeoff than any accuracy table.

2. Trees win partly by ignoring things. The greedy split search never selects a useless column, so uninformative features cost a tree nothing and cost a dense layer both capacity and samples. Real tabular datasets are mostly weak features, and the ability to not spend capacity on them is a structural advantage.

3. Speed is an accuracy multiplier, and benchmarks hide it. Equal-trial comparisons systematically understate the tree advantage, because in the same wall-clock time a tree explores a hundred configurations and a network explores five. Under the budget that actually constrains a project, engineer-hours, the gap is larger than any paper reports.

4. The most useful finding from the horse race is that both families should be in the ensemble. Every deep model lost individually to XGBoost and their ensemble with XGBoost beat XGBoost alone. Partly uncorrelated errors are worth more than a winner.

5. The extrapolation limit is structural and is the one thing tuning never fixes. A tree predicts constants on boxes. Any target with a trend must be differenced or detrended before the model sees it, and treating a flat forecast as a hyperparameter problem wastes weeks.

6. Preprocessing can hand away the tree's advantage. Standardising and applying PCA turns axis-aligned structure into diagonal structure, which is precisely the rotation intervention that collapses tree performance. On tabular data, "clean up the features first" is not a neutral step.

7. Read the baseline's tuning protocol before the results table. The single most reliable predictor of whether a new tabular architecture's claimed win survives replication is whether the baseline received comparable tuning effort. Its absence is disqualifying.

8. The comparison dissolves rather than resolves. The likeliest end to this question is not a tabular architecture that beats GBDT on spreadsheets, but tabular problems acquiring text, image and pretrained components until the pure-tabular case stops being the one that matters.

Open Questions

Does the sample-size ceiling on prior-fitted networks move? TabPFN's reported wins are bounded at roughly 10,000 samples, a limit arising from performing inference in-context over the training set. Whether the approach extends by orders of magnitude, through architectural changes to the context mechanism or through retrieval over a larger training set, is genuinely open. It is measured that the method is strong below its ceiling; it is speculation that the ceiling is soft.

Is there a tabular pretraining objective that transfers? Language modelling transfers because text shares structure across corpora. Two tabular datasets share no columns, so what a pretrained tabular model would carry over is unclear. TabPFN's answer, pretraining on synthetic datasets drawn from a prior over data-generating processes, is one proposal and its generality is untested outside the regimes reported.

How much of the remaining gap is optimisation rather than architecture? The smoothness bias attributed to neural networks comes partly from SGD's implicit regularisation and weight decay, not solely from the function class. Whether a different optimiser or regularisation scheme narrows the gap on irregular targets is an open empirical question, and the Inria interventions localise the phenomenon without attributing it.

Are the benchmarks measuring the right thing? Every study here uses curated collections with cleaned features and stable distributions. Real tabular problems have leakage risk, temporal drift, missingness that changes meaning over time, and features whose availability at prediction time differs from training. It is plausible, and untested, that the robustness of each family under those conditions differs more than their accuracy on clean benchmarks does.

Does the ensemble result generalise? One controlled study found GBDT plus deep models beating GBDT alone. Whether that holds across regimes, and how the gain compares against simply spending the same effort on features, has not been established at the scale the individual-model comparison has.

Sources and Further Reading

1. Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). "Why do tree-based models still outperform deep learning on tabular data?" NeurIPS 2022 Datasets and Benchmarks Track. arXiv:2207.08815
2. Shwartz-Ziv, R., & Armon, A. (2022). "Tabular Data: Deep Learning is Not All You Need." Information Fusion, 81, 84-90. arXiv:2106.03253
3. Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). "Revisiting Deep Learning Models for Tabular Data." NeurIPS 2021. arXiv:2106.11959
4. Hollmann, N., Müller, S., Purucker, L., et al. (2025). "Accurate predictions on small data with a tabular foundation model." Nature, 637(8045). Nature
5. Gorishniy, Y., Kotelnikov, A., & Babenko, A. (2024). "TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling." arXiv:2410.24210
6. Chen, T., & Guestrin, C. (2016). "XGBoost: A Scalable Tree Boosting System." KDD '16, 785-794. arXiv:1603.02754
7. Ke, G., Meng, Q., Finley, T., et al. (2017). "LightGBM: A Highly Efficient Gradient Boosting Decision Tree." NeurIPS 2017. NeurIPS Proceedings
8. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). "CatBoost: unbiased boosting with categorical features." NeurIPS 2018. arXiv:1706.09516
9. Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine." Annals of Statistics, 29(5), 1189-1232. DOI
10. Breiman, L. (2001). "Random Forests." Machine Learning, 45(1), 5-32. DOI
11. Lundberg, S. M., Erion, G., Chen, H., et al. (2020). "From local explanations to global understanding with explainable AI for trees." Nature Machine Intelligence, 2, 56-67. DOI
12. Arik, S. Ö., & Pfister, T. (2021). "TabNet: Attentive Interpretable Tabular Learning." AAAI 2021. arXiv:1908.07442

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.