The Inductive Bias Mismatch on Tabular Data

Deep learning displaced everything else on images, audio and text, and did not displace gradient-boosted trees on tabular data. This is not because tabular problems are easy or because nobody tried. It is because the properties that make neural networks work on perceptual data are absent from tables, and the properties tables do have are ones trees exploit directly.

The three mismatches

Target functions are irregular. A neural network is biased toward smooth functions: it interpolates, and it generalises by assuming nearby inputs have nearby outputs. Tabular targets frequently are not smooth. A threshold at exactly 65 years of age, a rule that applies only above a specific balance, a categorical interaction with no ordering, are all step functions with sharp boundaries. Trees represent them exactly with a split; a network approximates them with a sigmoid and needs many units and much data to sharpen it.

Features are heterogeneous and unordered. Column three has no relationship to column four, and permuting the columns should not change anything. Convolutions exploit spatial adjacency and transformers exploit sequence structure, and neither exists here. A multilayer perceptron is permutation-equivariant over inputs only in the sense that it has to learn every relationship from scratch with no structural prior at all.

Uninformative features are common and harmful. Real tables contain many columns that carry nothing, and a network is rotationally invariant in its first layer, meaning it mixes all features immediately and cannot cheaply ignore one. A tree simply never splits on a useless feature and pays nothing for it, which is a decisive advantage when half the columns are noise.

Grinsztajn and colleagues isolated these three empirically, showing that removing uninformative features narrows the gap and that smoothing the target function narrows it further (Grinsztajn et al., 2022, arXiv:2207.08815).

What follows for design

The productive response has not been bigger networks but incorporating the missing biases. Learned feature embeddings per column give categorical variables a representation without forcing an ordering. Attention over columns lets the model select rather than mix everything. Explicit feature selection layers restore the ability to ignore. Architectures that model piecewise-constant functions directly attack the smoothness mismatch.

The results are respectable and the honest summary remains that gradient-boosted trees are the correct default for a tabular problem of moderate size, and that a neural approach needs a specific reason.

When it breaks

The mismatch weakens with scale. With millions of rows, a network has enough data to learn sharp boundaries and to identify useless features. The tree advantage is largest in the small and medium regime, which is where most tabular problems live.

Very high cardinality favours networks. A categorical variable with a million levels is awkward for a tree, which must partition its levels, and natural for an embedding, which learns a vector per level with sharing. This is why recommendation and advertising, dominated by high-cardinality identifiers, moved to networks while general tabular problems did not.

Multimodality changes the calculation. A problem combining tabular features with text or images needs a network for the unstructured parts, and once one exists, handling the tabular columns inside it can beat maintaining two models and combining them.

Benchmark selection drives the conclusion. Papers claiming neural superiority and papers claiming tree superiority frequently use different dataset collections with different characteristics, and the disagreement is largely about which distribution of problems is representative.