Categorical Encoding

A postcode column with 28,000 distinct values. One-hot encoding produces 28,000 sparse columns, most of which appear in a handful of rows. Label encoding produces one column in which postcode 4,102 is numerically between 4,101 and 4,103, implying an ordering the model will happily exploit and that does not exist. Neither is right, and the choice between the alternatives is where most of the modelling decision lives.

The three families

One-hot creates one indicator per level. It is correct, assumption-free, and the only option that says nothing about the relationship between levels. It fails on cardinality, both in width and in statistical support: a level appearing five times gets a coefficient estimated from five observations. Dropping one level avoids exact collinearity in a linear model and is irrelevant for a tree.

Ordinal or label encoding maps levels to integers. It is correct when the categories genuinely have an order (small, medium, large; a plan tier) and a serious misspecification otherwise, since a linear model reads it as a numeric scale. For trees the harm is smaller: the model can carve the integer axis into arbitrary sets of contiguous ranges, so an arbitrary ordering costs splits rather than correctness. That is why "just label-encode it for LightGBM" is common advice that is defensible in a way it would not be for a regression.

Target encoding replaces a level with a statistic of the target computed over rows in that level, usually the mean. It compresses any cardinality into one dense column carrying the information the model needs, which is why it is standard in competitive tabular work. It also leaks the target into a feature, which is the entire difficulty.

Making target encoding survive

The naive version, computing the mean over all training rows including the current one, is direct leakage: for a rare level with a single row, the encoded value simply is the label. Two mechanisms are required.

Smoothing toward the global mean. With \(n_c\) rows in category \(c\), class mean \(\bar{y}_c\) and global mean \(\bar{y}\), a standard form is

which is a Beta prior with strength \(m\) in exactly the pseudo-observations sense. Rare levels are pulled to the global mean; common levels keep their own. \(m\) is the one hyperparameter and it directly controls how much a thin category is trusted.

Out-of-fold computation. Encode each training row using target statistics computed from the other folds only, so no row contributes to its own encoding. CatBoost generalises this with ordered target statistics, using a random permutation so each row's statistics come only from rows earlier in the permutation, which removes the prediction shift that the fold-based scheme leaves at fold boundaries (Prokhorenkova et al., 2018, NeurIPS, arXiv:1706.09516).

Both are needed. Smoothing without out-of-fold still leaks; out-of-fold without smoothing still gives noisy encodings for rare levels.

When it breaks

Unseen categories at serving time. One-hot silently produces an all-zero row, which the model interprets as "none of the known levels", a defensible default. Target encoding has to fall back to the global mean, and that fallback must be an explicit, tested code path rather than a KeyError in production. New postcodes, new device models and new merchant IDs appear constantly, and the rate at which they appear is worth measuring.

Cardinality drift. A category with 200 levels at training time and 2,000 at serving time is a different feature. Monitoring the count of distinct levels and the fraction of traffic falling into unseen levels catches this before the metric does.

Target encoding bakes in the training-period base rate. If the overall positive rate shifts, every encoded value is now miscalibrated in a correlated way, which is worse than a single drifting feature because it moves the whole column at once.

Hashing trades collisions for a fixed width. The hashing trick maps levels into \(2^k\) buckets, bounding memory and handling unseen levels automatically at the cost of unrelated levels sharing a bucket. It is the right answer for genuinely open-ended vocabularies at scale, and it makes the model uninspectable at the feature level, which is a real cost when someone asks why a prediction was made.

Learned embeddings need volume. Mapping levels to dense trainable vectors is what large-scale recommenders do, and it is what makes deep learning competitive there. It requires enough data per level to learn anything, which is why it wins at hundreds of millions of rows and loses at ten thousand.