Numerical Feature Embeddings for Tabular Models

On the California Housing regression task, an ensemble of plain MLPs reaches an RMSE of 0.486. CatBoost reaches 0.430. Keep the MLP backbone, change only how each numerical input is represented before the first layer, retune as usual, and the ensemble reaches 0.433 (Gorishniy, Rubachev & Babenko, 2022, On Embeddings for Numerical Features in Tabular Deep Learning, NeurIPS, arXiv:2203.05556). The authors report this as the first time deep models performed on par with gradient-boosted trees on California Housing and Adult. The bottleneck was the input encoding.

Attention over columns mentions these encodings in passing and notes that the encoding choice can matter more than the backbone. This concept is the mechanism behind that claim.

Why a scalar input is a poor start

A standard MLP multiplies each numerical feature by a weight and adds it to everything else in the first layer. The feature enters as a single direction in activation space, and any sharp, local behaviour, such as a price effect that switches on above 500,000, has to be manufactured by many ReLUs downstream.

Networks are bad at manufacturing it. Deep ReLU networks show a spectral bias toward low-frequency functions, learning smooth global trends before local variation (Rahaman et al., 2019, On the Spectral Bias of Neural Networks, ICML, arXiv:1806.08734). Tabular targets are often the opposite: step-like thresholds that trees represent with one split. This is one strand of the inductive bias mismatch on tabular data.

In coordinate-based vision the same problem was solved by transforming inputs before the network sees them: passing low-dimensional coordinates through Fourier features lets an MLP learn high-frequency functions (Tancik et al., 2020, Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS, arXiv:2006.10739). Gorishniy and colleagues applied the idea per column. Each numerical feature \(x_i\) is mapped independently to a vector \(z_i = f_i(x_i) \in \mathbb{R}^{d_i}\), with no parameters shared between features, and the vectors are concatenated for an MLP or passed as tokens to a transformer.

Piecewise linear encoding

Split a feature's range into \(T\) bins \(B_t = [b_{t-1}, b_t)\). The piecewise linear encoding is \(\mathrm{PLE}(x) = [e_1, \dots, e_T]\) with

Bins entirely below \(x\) are full, bins above are empty, and the bin containing \(x\) is filled fractionally. It is a continuous, ordered cousin of one-hot encoding.

Take bin edges \(0, 10, 20, 50, 100\) and \(x = 35\). The first two bins lie below 35, so \(e_1 = e_2 = 1\). The third bin is \([20, 50)\) and \(x\) is halfway through it: \(e_3 = (35 - 20)/30 = 0.5\). The fourth is empty. \(\mathrm{PLE}(35) = [1, 1, 0.5, 0]\). Moving \(x\) from 35 to 36 changes one coordinate slightly; moving it across 50 starts filling a new coordinate, which a following layer can weight differently.

Adding a linear layer gives each bin its own learned vector: \(f(x) = v_0 + \sum_t e_t v_t\). That is what lets the model place a sharp change at a bin edge.

Bins come from one of two recipes. Quantile bins use empirical quantiles of the training data, unsupervised and robust to skew. Target-aware bins grow a decision tree on that single feature against the label and use its leaves, so edges land where the target actually changes. On the paper's GBDT-friendly benchmark target-aware bins tended to win, and the authors explicitly decline to generalise that to other data.

Periodic embeddings

The second family adapts Fourier features:

with frequencies \(c_j\) initialised from \(\mathcal{N}(0, \sigma)\) and, unlike Tancik et al., trained. The scale \(\sigma\) sets the range of frequencies and is the hyperparameter the authors single out as important to tune. Followed by a linear layer and ReLU, this is the PLR embedding. In their ensemble comparison MLP-PLR had the best average rank of any model, 3.0 against CatBoost's 3.6, and MLPs with good embeddings became competitive with transformer backbones.

The disagreement worth knowing

The benchmark was deliberately tilted toward problems where trees are strong, as the authors state. That cuts both ways: it makes the closing of the gap more impressive and makes the relative ranking of embedding types less transferable. Average ranks across eleven datasets also hide large per-dataset variance; on some pairs of backbone and dataset the gap to boosting did not close. The honest reading is that numerical embeddings are one of the highest-value, lowest-risk changes to a tabular network, not that they settle the tree-versus-network question.

When it breaks

Parameters grow with features. Each feature's own linear layer over its encoding has \(T \times d\) weights plus a bias, so a table with 500 numerical columns and 64 bins at \(d = 32\) adds about a million parameters before the backbone.

Target-aware bins can leak and overfit. Building bins with labels on anything but the training split leaks, and on small data a tree fitted per feature memorises noise, placing edges that do not generalise.

Periodic embeddings are sensitive to \(\sigma\). Too small and the embedding is nearly linear; too large and it is high-frequency noise that overfits.

Extrapolation is linear, not learned. Values beyond the last training edge push \(e_T\) above 1 and extend the final segment linearly, which may be badly wrong under distribution shift.

Ties collapse bins. Features with heavy repeated values, such as zeros in counts, produce zero-width quantile bins that are removed, leaving far fewer effective bins than configured.