Benchmarking Tabular Methods Honestly

The tabular literature contains a long-running disagreement about whether neural networks beat gradient-boosted trees, with papers on both sides reporting clear results. The disagreement is not primarily about the models. It is about the benchmarks, the tuning and the aggregation, and understanding those explains most of the contradiction.

Where the disagreement comes from

Dataset selection. There is no canonical tabular benchmark suite, so each paper assembles a collection. Collections differ in size, feature count, categorical proportion, class balance and how much irrelevant signal they contain, and each of those shifts the answer. A collection of large datasets with many high-cardinality categoricals favours networks; a collection of medium datasets with many uninformative columns favours trees.

Tuning budget asymmetry. This is the largest single factor. A paper proposing a neural architecture tunes it extensively and compares against a boosted tree with default or lightly tuned parameters. Since boosted trees are highly sensitive to a handful of parameters, learning rate, tree count, depth, regularisation, an untuned baseline is substantially below its achievable performance. Reported gaps frequently close entirely under equal tuning budgets.

Preprocessing differences. Categorical encoding, missing value handling and numerical scaling affect the two families differently, and a pipeline tuned for one is a handicap for the other.

Aggregation across datasets. Mean rank, mean normalised score and count of wins can order methods differently on the same results, and the choice is rarely justified.

What a comparison needs

Equal tuning budgets, stated in trials or in compute, applied to every method including the baselines. Multiple random seeds with variance reported, since the difference between methods is often smaller than the difference between seeds. Per-dataset results rather than only an aggregate, so a reader can see whether a method wins broadly or on a subset. Dataset characteristics reported, since the conclusion is conditional on them. And an explicit statement of the aggregation rule.

Adding the training and inference cost makes the comparison decision-relevant, since a neural model matching a boosted tree while taking twenty times longer to train is not a better default even where the accuracy is equal.

When it breaks

Aggregate rankings hide conditional structure. The useful finding is usually not which method is better but on which kind of dataset each wins, and a single ranking discards exactly that. Reporting performance against dataset properties is more informative than a leaderboard position.

Public datasets are heavily overfitted. Widely used tabular datasets have been used for hyperparameter selection across hundreds of papers, so reported performance on them is optimistic for everyone and the ranking is partly a record of who tuned against them longest.

The realistic setting is rarely tested. Practitioners face temporal splits, distribution shift, missing values arriving at inference, and a training budget of hours. Benchmarks use random splits on clean static data, and results under those conditions do not transfer straightforwardly to the ones that obtain.

Operational properties are not benchmarked at all. Training time, inference latency, memory, robustness to a missing column, and how the model degrades under shift are what decide a production choice, and none of them appear in an accuracy table.