Feature Scaling and Transformations

A 15-nearest-neighbour classifier on six synthetic features scores 92% accuracy. Multiply one feature by 1,000, as happens when one column arrives in grams and the rest in kilograms, and accuracy drops to 59%, barely above a coin flip. (Both figures come from a small scikit-learn simulation.) The information in the data is identical. What changed is that Euclidean distance now measures almost nothing but that one column. A gradient-boosted tree trained on the same two versions would produce the same splits and the same predictions. Whether scaling matters is not a property of the data; it is a property of the model.

Which models care, and why

Distance-based methods (k-nearest neighbours, k-means, DBSCAN, RBF-kernel SVMs) sum squared differences across features, so each feature's weight is its variance. Scaling is an implicit choice of which dimensions count equally, as the k-means concept on the unsupervised track notes.

Penalised linear models care through the penalty. Ordinary least squares is scale-equivariant: rescale a feature by \(c\) and its coefficient becomes \(\beta/c\) with identical predictions. Ridge and lasso penalise \(\sum_j \beta_j^2\) or \(\sum_j |\beta_j|\), and shrinking a feature's scale inflates the coefficient it needs, so it gets penalised harder. Unscaled, the penalty targets features by their units. Some libraries standardise internally and report coefficients on the original scale; others do not, and the difference silently changes which features a lasso keeps.

Gradient-based training cares through conditioning. For least squares the loss curvature is governed by \(X^\top X\), and a feature with scale 1,000 contributes curvature about \(10^6\) times larger along its direction. The largest stable learning rate is set by the steepest direction, so the others crawl. Neural networks inherit this at the input layer, which is why input normalisation is standard even with normalisation layers inside the network.

Tree ensembles split on thresholds and depend only on the ordering of values, so any strictly monotone transform leaves exact trees unchanged. Histogram-based gradient boosting bins values first and is approximately invariant.

The transforms

Standardisation, \(z = (x - \hat\mu)/\hat\sigma\), gives zero mean and unit variance and is the default. Min-max scaling maps to \([0,1]\) using the observed extremes, so a single outlier at \(10^6\) compresses every other value into a sliver near zero. Robust scaling uses the median and interquartile range and resists that. None of these change the shape of the distribution; a right-skewed feature stays right-skewed.

Power transforms do change shape. The Box-Cox family, for strictly positive \(x\) (Box and Cox, 1964, An Analysis of Transformations, JRSS-B 26(2)), is

with \(\lambda\) chosen by maximum likelihood under an assumption that the transformed values are normal. \(\lambda = 1\) is a shift, \(\lambda = 0.5\) a square root, \(\lambda = 0\) the log. Yeo-Johnson extends the idea to zero and negative values by applying a Box-Cox-like transform of \(x+1\) for \(x \ge 0\) and a mirrored transform of \(-x+1\) with power \(2-\lambda\) for \(x < 0\) (Yeo and Johnson, 2000, A new family of power transformations to improve normality or symmetry, Biometrika 87(4)).

Quantile transforms map each value through the empirical CDF and then, optionally, the inverse normal CDF: \(z = \Phi^{-1}(\hat F_n(x))\). The output is exactly uniform or normal on the training data whatever the input shape, and outliers are neutralised. The price is that distances between values are discarded beyond their ranks, and unseen extremes at inference are clipped to the edge of the training range.

For neural networks on tabular data, even these are not the end point. Learned encodings of numerical features, such as piecewise linear encodings over quantile bins, can let a plain MLP compete with much larger architectures (Gorishniy, Rubachev and Babenko, 2022, On Embeddings for Numerical Features in Tabular Deep Learning, NeurIPS, arXiv:2203.05556).

Transforming the target is a different decision

Log-transforming a skewed target such as revenue often improves fit, and it changes the estimand. A model minimising squared error on \(\log y\) estimates \(\mathbb{E}[\log y \mid x]\), and exponentiating gives something closer to the conditional median than the mean. If \(\log y \mid x\) is normal with standard deviation \(\sigma\), then \(\mathbb{E}[y \mid x] = \exp(\mu + \sigma^2/2)\). With \(\sigma = 1\) the naive back-transform \(\exp(\mu)\) is only \(e^{-0.5} \approx 61\%\) of the mean, a 39% underestimate that sums to a large miss in any aggregate forecast. Practitioners disagree on the remedy: a parametric correction assumes the residual distribution, a smearing estimator uses the empirical residuals, and many argue for modelling on the original scale with a loss or likelihood suited to skewed positive data, such as Poisson, gamma or Tweedie, instead of transforming at all.

When it breaks

Scalers are fitted models and leak like models. Computing \(\hat\mu\) and \(\hat\sigma\) on the full dataset before splitting lets test statistics shape training features. The fix costs nothing: fit inside the cross-validation pipeline, as the target leakage concept on this track describes.

Training statistics are frozen into serving. A scaler fitted in January encodes January's mean. If the feature drifts, serving inputs are standardised against the wrong centre, and if the serving path recomputes statistics per batch, it is a textbook source of train-serve skew.

Centring destroys sparsity. Subtracting the mean from a sparse count matrix turns every zero into a non-zero, which can multiply memory by orders of magnitude. Scale without centring for sparse inputs.

Box-Cox assumes something to normalise toward. Bimodal or zero-inflated features have no power transform that makes them normal, and the likelihood-optimal \(\lambda\) can be extreme and unstable across resamples. A missingness-style indicator plus a log often does more.