Attention Over Columns
How transformer architectures are adapted to tables, what per-feature embeddings buy, and why the attention mechanism addresses the feature selection problem specifically.
The transformer's contribution to tabular deep learning is not sequence modelling, since a row has no sequence, but the ability to select. Attention lets a model attend to a few features and ignore the rest, which is precisely the capability a multilayer perceptron lacks and trees have natively.
The construction
Each feature is embedded into a shared vector space. Categorical features get a lookup table per column, which handles arbitrary cardinality and no ordering. Numerical features get a learned projection, or a more expressive encoding: piecewise-linear encodings over learned bins, or periodic features, both of which help the model represent sharp thresholds that a linear projection cannot.
The row becomes a set of feature tokens, and self-attention runs over them. Since there is no order, no positional encoding is used, and the model is permutation-equivariant by construction, which matches the data's actual symmetry.
TabTransformer applies this to categorical features while passing numerical ones through separately. FT-Transformer embeds all features uniformly and adds a classification token whose final representation is the prediction, and it is the stronger and simpler design, standing as the main neural baseline in most comparisons.
What attention provides
Sample-conditional feature selection. The attention weights depend on the row, so the model can attend to different features for different examples. A tree makes a global structural choice about which features to split on; attention makes a per-example one, which is a genuine capability trees do not have.
Interaction modelling without enumeration. Pairwise attention captures feature interactions directly, where a tree represents an interaction through a sequence of splits and needs depth and data for each one.
Interpretability, with caveats. Attention weights indicate which features the model looked at, which is useful and is not a faithful explanation, since attention is one component among several and high attention does not imply high influence on the output.
When it breaks
Attention over columns is quadratic in feature count. A table with a thousand columns is expensive, and the cost buys interactions most of which do not exist. Sparse or grouped attention is required at that width.
Numerical embedding choice matters more than the architecture. How a continuous feature is encoded, plain projection, learned bins, or periodic features, moves results more than the choice between competing transformer variants, and it is the thing worth tuning first.
Regularisation and training recipe carry much of the reported gain. Careful tuning of a plain MLP closes much of the gap to specialised architectures, and comparisons that tune the new model and not the baseline overstate the architectural contribution. This is a documented pattern in the tabular literature specifically.
Small datasets remain unfavourable. These models have many parameters relative to a few thousand rows, and no architectural bias fully compensates for a data regime where a boosted tree with strong regularisation is simply better matched.
12 flashcards for this concept
Click a card to reveal the answer.