UMAP, t-SNE and What They Distort

A two-dimensional embedding shows six well-separated blobs, two of them touching, one far off to the right. The natural reading is that there are six groups, two of which are related, and one that is very different. All three of those inferences can be false, and which of them is false depends on hyperparameters nobody reported.

Neighbour embeddings optimise one thing: that points close in the original space stay close in the picture. They make no promise at all about the distance between things that were far apart, and that omission is where every misreading comes from.

What t-SNE actually minimises

t-SNE converts distances to probabilities in both spaces and matches them (van der Maaten and Hinton, 2008, Visualizing Data using t-SNE, JMLR 9). In the high-dimensional space, \(p_{j|i}\) is a Gaussian centred on point \(i\), with its bandwidth chosen per point so the distribution has a fixed perplexity, the user's main knob and an effective neighbour count. In the low-dimensional space, \(q_{ij}\) uses a Student-t kernel with one degree of freedom. The objective is \(\mathrm{KL}(P \,\|\, Q)\).

Two design choices drive the behaviour. The heavy-tailed t-kernel in 2D lets moderately distant points sit much further apart than a Gaussian would, which is the fix for the crowding problem: there is simply not enough room in two dimensions for the neighbourhood structure of a high-dimensional space, and something has to give.

The asymmetric KL is the more consequential choice. \(\mathrm{KL}(P\|Q) = \sum p \log(p/q)\) imposes a large penalty when \(p\) is large and \(q\) is small, meaning nearby points placed far apart, and almost no penalty when \(p\) is small and \(q\) is large, meaning distant points placed close together. The objective is therefore explicitly indifferent to preserving large distances. Global structure is not distorted by accident; it is not in the loss.

UMAP frames the same task as optimising a fuzzy topological representation and uses a cross-entropy objective with negative sampling, which gives it better runtime and somewhat more global structure retention (McInnes, Healy and Melville, 2018, arXiv:1802.03426). The qualitative caveats below apply to both.

Reading a plot correctly

Three things in a neighbour embedding carry no reliable meaning.

Cluster size. The area a group occupies reflects local density in the original space and the algorithm's spreading behaviour, not the group's variance or its member count. A tight, numerous cluster can render smaller than a sparse, small one.

Distance between clusters. Because the objective barely penalises misplacement of distant points, the gap between two blobs is not a dissimilarity. Two adjacent clusters need not be more similar than two far-apart ones.

Apparent separation. t-SNE and UMAP will produce visually distinct clusters from uniform random noise at some hyperparameter settings. Separation in the picture is evidence about the algorithm's behaviour at those settings, not evidence that clusters exist.

Chari and Pachter document the extent of this distortion quantitatively for single-cell genomics, where these plots are near-universal, and find extensive distortion together with inconsistent practice in how the embeddings are produced and read (Chari and Pachter, 2023, PLOS Computational Biology 19(8): e1011288).

When it breaks

Hyperparameters change the answer, not just the aesthetics. Perplexity in t-SNE and n_neighbors in UMAP set the scale at which neighbourhoods are defined; low values fragment continuous structure into apparent clusters, high values merge genuine ones. Any published embedding without its parameters is unreproducible, and running two or three settings is the minimum honest practice.

There is no out-of-sample map for t-SNE. The embedding is a set of optimised coordinates, not a function. New points require refitting, and the new layout is not comparable to the old one. UMAP does provide a transform for new points, which is a genuine practical advantage and one of the main reasons it displaced t-SNE in pipelines.

Randomness is not incidental. Different seeds give different layouts. Structure that survives across seeds is worth attention; structure that does not is not.

Downstream clustering on the embedding is circular. Running k-means on 2D UMAP coordinates clusters the picture, not the data, and inherits every distortion above. Cluster in the original or a PCA-reduced space, then use the embedding to display the labels, which also gives an honest visual check on whether the clustering agrees with local structure.

Neither is a dimensionality reduction step for a model. Both are visualisation tools. Their output coordinates are not features; they are non-linear, unstable under refitting, and optimised for a criterion unrelated to any downstream task.