Prior-Fitted Networks and In-Context Tabular Learning
How a model trained entirely on synthetic datasets can classify a real one without fitting, what approximating the Bayesian posterior in a forward pass means, and the constraints that follow.
The usual pipeline fits a model to a dataset. Prior-fitted networks do something else: a transformer is trained once, on millions of synthetic datasets generated from a prior over data-generating processes, to predict held-out labels given a labelled context. At inference the real dataset is placed in the context window and predictions are read out in a single forward pass. No fitting occurs.
What is being learned
The training objective is to predict the label of a query point given a set of labelled examples, across millions of tasks drawn from the prior. The optimal solution to that objective is the posterior predictive distribution under the prior, so a network trained to convergence approximates Bayesian inference over the space of data-generating processes the prior describes.
TabPFN implements this with a prior based on structural causal models, generating synthetic datasets whose structure resembles what tabular data plausibly looks like (Hollmann et al., 2023, arXiv:2207.01848). On small numerical classification problems it was competitive with tuned gradient-boosted trees while requiring no tuning and running in under a second, which is a different point on the cost-quality curve rather than an incremental improvement.
Why this is interesting beyond the benchmark
The prior is explicit and modifiable. Where conventional models embed their assumptions in an architecture and a regulariser, a prior-fitted network's assumptions are the generator, which can be inspected and changed. Encoding domain knowledge becomes a matter of generating datasets that reflect it.
Amortisation is the other property. All the computation happens once, during pretraining, and inference is a forward pass. For a workflow fitting thousands of small models, this inverts the cost structure entirely.
The constraints
The original version was limited by the context window and by the training distribution: roughly a thousand rows, around a hundred features, up to ten classes, numerical features, and classification only. Later versions have extended these substantially, and the constraint is structural rather than incidental, since the dataset must fit in the context.
Performance also degrades as a dataset departs from the prior. A problem whose structure the generator does not produce is one the model has no posterior for, and unlike a fitted model it cannot adapt.
When it breaks
The prior is the model. Everything the network knows comes from the synthetic generator, so a domain whose structure the prior does not cover is a domain where the approach has nothing to offer, and diagnosing that requires understanding the prior rather than the network.
Context length bounds dataset size directly. Beyond the window, subsampling or ensembling over subsets is required, and both discard information in ways a fitted model does not.
Interpretation is harder, not easier. There are no learned parameters to inspect and no feature importances in the usual sense, so post-hoc attribution is the only route and it explains a forward pass rather than a fitted relationship.
Evaluation must use datasets outside the prior's construction. Testing on data resembling the generator's output measures how well the network learned its own prior, which is a different question from how well the prior matches reality.
12 flashcards for this concept
Click a card to reveal the answer.