Why Trees Still Beat Deep Nets on Tabular Data

Deep learning displaced every prior method on images, audio and text within a few years of arriving. On tabular data it has been trying since roughly 2016 and has not managed it. Grinsztajn, Oyallon and Varoquaux benchmarked standard and recent deep architectures against XGBoost and random forests across 45 curated tabular datasets, with a hyperparameter search budget of 20,000 compute hours per learner, and found tree-based models remain state of the art on medium-sized data of around 10,000 samples, before accounting for their large speed advantage (Grinsztajn et al., 2022, NeurIPS Datasets and Benchmarks, arXiv:2207.08815).

The interesting part of that paper is not the leaderboard. It is the empirical investigation of why, which produced three concrete challenges for anyone building tabular networks.

The three inductive biases

Neural networks are biased toward smooth functions. Their empirical procedure was to smooth the target function of a real dataset with a Gaussian kernel and re-evaluate: tree performance degrades markedly, neural network performance barely moves. Tabular targets are frequently irregular, with genuine discontinuities at thresholds that carry business meaning (a credit limit, an age boundary, a plan tier). A tree represents a step with one split. An MLP must approximate it with many units and, under weight decay and SGD's implicit bias, prefers not to.

Uninformative features hurt MLPs disproportionately. Removing features in increasing order of importance closes much of the gap; adding uninformative features widens it. A tree's greedy split search simply never selects a useless column, so its capacity is not consumed by it. A dense layer assigns every input a weight and must learn to zero it, which costs both samples and effective capacity. Real tabular datasets are full of near-useless columns, and unlike pixels, dropping them is a legitimate modelling step.

Rotation invariance is the wrong prior. MLPs are rotationally invariant in the sense that training on a randomly rotated feature space yields the same learned function up to that rotation; the paper confirms performance is unchanged under rotation while tree performance collapses. But in tabular data a column means something. "Age" and "income" are not an arbitrary basis, and a method that treats them as one has discarded genuine information about which directions matter. Being non-rotation-invariant is an advantage precisely because the given basis is meaningful.

What else the comparison is measuring

Trees also carry practical advantages that are easy to under-weight in an accuracy table. They need no scaling, no normalisation, and no skew correction, since splits depend only on the ordering of values. They handle mixed types and high-cardinality categoricals without embedding design. They train in seconds to minutes on a laptop CPU, so the effective hyperparameter search per unit of engineer time is far larger. Under equal wall-clock budgets rather than equal trial counts, the gap widens further.

The honest counterweight is that this comparison is about pure tabular prediction. As soon as a problem includes text fields, images, or a need to co-train with another modality, gradient descent on a shared representation is doing something no tree can, and the comparison stops being the relevant one.

Where the ordering has flipped

The literature is not static, and two developments are worth tracking precisely.

TabPFN takes a different route entirely: a transformer pre-trained on large numbers of synthetic tabular datasets performs Bayesian inference in-context, with the training set supplied as the prompt and no gradient steps at fit time. The published claim is that it outperforms prior methods on datasets up to around 10,000 samples using substantially less training time (Hollmann et al., 2025, Accurate predictions on small data with a tabular foundation model, Nature 637). The size ceiling is the caveat that matters: this is a small-data result.

Ensembled MLPs have narrowed the gap through engineering rather than architecture, with parameter-efficient ensembling reported to make simple MLPs competitive with GBDT on standard benchmarks (Gorishniy et al., 2024, TabM, arXiv:2410.24210).

Neither settles the question, and benchmark curation is itself contested: results depend heavily on which datasets are included, how categoricals are encoded, and whether the tuning budget is measured in trials or in hours.

When it breaks

"Tabular" is not one regime. The tree advantage is documented most clearly at thousands to tens of thousands of rows. At hundreds of millions of rows with high-cardinality categoricals, the deep learning story is far stronger, which is why large-scale recommender and ads systems are dominated by embedding-based networks rather than by GBDT.

Trees cannot extrapolate. Any target with a trend outside the training range gets the nearest leaf's constant. This is a hard structural limit, not a tuning failure, and it is why forecasting pipelines difference the series before handing it to a tree.

No transfer, no pretraining, no multimodality. A tree ensemble learns one dataset from scratch each time. Where you have a related large corpus, an image column, or free text, the ability to share a representation is worth more than the tabular inductive bias.

Benchmark results transfer poorly to one dataset. An aggregate result across 45 datasets says nothing binding about yours. The cost of settling it directly, one XGBoost run against one tuned MLP, is measured in hours, and that is a better use of time than reasoning from the literature.