Single-Cell Foundation Models

A single-cell RNA sequencing experiment gives you a matrix: tens of thousands of cells by twenty thousand genes, each entry a noisy count of how many transcripts of that gene were captured in that cell. The analogy to language modelling is seductive. A cell is a sentence, genes are tokens, expression level is something like position or frequency, and there are hundreds of millions of published cells to pretrain on. Two systems took the analogy seriously at scale: Geneformer, pretrained on Genecorpus-30M, roughly 30 million human single-cell transcriptomes (Theodoris et al., 2023, Transfer learning enables predictions in network biology, Nature 618, 616–624), and scGPT, pretrained on over 33 million cells (Cui et al., 2024, scGPT, Nature Methods 21, 1470–1480).

How a cell becomes a sequence

The tokenisation choice is the interesting part, and the two systems make it differently. Geneformer converts each cell into a rank ordering: genes sorted by expression normalised against their median across the corpus, so the input is a ranked list of gene identifiers with no explicit magnitudes. This deliberately discards absolute counts, which are dominated by technical variation in sequencing depth. scGPT instead keeps binned expression values alongside gene identifiers and trains a masked objective adapted to non-sequential data: rather than predicting a masked token from left context, it predicts masked gene expression from the observed genes in the same cell, iteratively.

Both then claim the standard foundation-model payoff: fine-tune on a small labelled dataset for cell type annotation, batch integration, perturbation response prediction, or gene network inference, and beat task-specific models trained from scratch.

The benchmark that complicated the story

Then came a careful zero-shot evaluation. Kedzierska and colleagues asked what the pretrained representations are worth before fine-tuning, on cell type clustering and batch effect correction, and found both Geneformer and scGPT frequently underperforming simple established baselines — selecting highly variable genes, and standard integration methods like scVI and Harmony (Kedzierska et al., 2023/2025, Zero-shot evaluation reveals limitations of single-cell foundation models, bioRxiv 2023.10.16.561085; published in Genome Biology). Related work on perturbation analysis reached a similarly deflating conclusion, with PCA remaining a strong competitor.

This does not show the models are useless; fine-tuned performance is a different measurement from zero-shot embedding quality, and the original papers evaluated the former. It does show that the phrase "foundation model" was carrying an assumption that had not been tested: that pretraining produces a general-purpose representation of cell state which transfers without supervision, the way a language model's embeddings do. On this evidence, for these models and these tasks, it does not.

When it breaks

• Batch effect is the confounder that eats everything. Public single-cell corpora are aggregations across labs, protocols, and platforms, and the largest source of variance is usually which lab produced the sample. A model can score well by learning the batch, and clustering metrics will reward it. Any evaluation that does not hold out entire studies is measuring the wrong thing.
• The tokenisation is a strong, unexamined prior. Rank ordering throws away magnitude; binning throws away resolution. Neither is obviously right, and results are sensitive to the choice in ways rarely ablated.
• Genes are not tokens in the way the analogy needs. Word order carries syntax; gene order in a cell carries nothing intrinsic, so the positional machinery a transformer brings has no natural referent and must be repurposed.
• Data scale has not behaved like a scaling law. More cells has not reliably meant better transfer, partly because added cells are often redundant tissue types rather than new biology (see ML for science pitfalls for the leakage-adjacent version of this problem).
• Zero-shot and fine-tuned claims get conflated in summaries. When you read that a single-cell foundation model "outperforms specialised methods", check which regime, which baselines, and whether the baselines were tuned with equal care. The gap between those two readings is where most of the disagreement in this field currently lives.