What FID Measures and What It Misses
The Gaussian assumption inside Frechet Inception Distance, the sample-size bias that makes numbers incomparable across papers, and why precision and recall metrics exist.
FID is the number that decides which generative model gets published. It is a single scalar summarising a comparison between two distributions of images, it correlates reasonably with human judgement, and it is computed in a way that quietly encodes several assumptions most readers of a results table never examine.
The computation
Take a set of real images and a set of generated images. Push both through a fixed Inception-v3 network and extract the 2048-dimensional pool3 activations. Fit a multivariate Gaussian to each set, giving means \(\mu_r, \mu_g\) and covariances \(\Sigma_r, \Sigma_g\). Report the Frechet distance between those two Gaussians:
Lower is better, zero means the two Gaussian fits are identical. The metric comes from Heusel et al., 2017, arXiv:1706.08500, where it replaced Inception Score by adding a comparison against real data rather than judging generated images alone.
Three assumptions are now visible. The Inception features are a good summary of perceptual content. The feature distributions are approximately Gaussian. And the first two moments capture what matters about them.
The sample-size bias
FID is biased upward at small sample sizes, and the bias is substantial. The covariance estimate from \(N\) samples in 2048 dimensions is noisy, and that noise inflates the trace term. Reported FIDs at 10,000 samples are systematically worse than the same model measured at 50,000, and the gap is large enough to reverse rankings between models.
This makes cross-paper FID comparison unreliable unless sample count, reference set, image resolution, resizing method, and Inception weights all match. They frequently do not. The resizing detail is not pedantry: different libraries' bilinear implementations produce measurably different FIDs for the same images, an effect documented in reproducibility work on the metric. Treating a published FID as a physical constant of a model is a mistake.
Quality and coverage are one number here
FID collapses two independent failure modes into one scalar. A model that produces beautiful samples covering half the distribution and a model that covers everything with mediocre samples can score identically. Since these call for opposite interventions, the metric is at its least useful exactly when a team most needs direction.
Precision and recall for generative models separate them (Kynkaanniemi et al., 2019, arXiv:1904.06991). Precision estimates the fraction of generated samples that fall within the support of the real data manifold; recall estimates the fraction of the real manifold covered by generated samples. Precision is sample quality, recall is coverage, and mode collapse shows up as high precision with low recall, which FID reports as a middling single number. Density and coverage are later refinements with better robustness to outliers.
When it breaks
Inception features are not neutral. The network was trained on ImageNet classification, so it is sensitive to object identity and comparatively insensitive to properties ImageNet does not require: faces at fine detail, text, medical imaging structure, artistic style. FID on a domain far from ImageNet is measuring something, but not necessarily what you care about. Domain-specific feature extractors, and CLIP-based variants such as CMMD which drops the Gaussian assumption in favour of a kernel distance, exist for this reason.
It cannot see memorisation. A model that reproduces training images exactly achieves an excellent FID. Nothing in the metric penalises copying, so FID must be paired with a nearest-neighbour or memorisation check before any claim about generalisation.
The Gaussian fit discards structure. Real feature distributions are multimodal. Matching mean and covariance is compatible with getting the modal structure wrong, which is exactly the failure mode of a mode-collapsed generator. The assumption is a convenience that makes the metric closed-form and cheap.
Guidance scale makes it a curve, not a point. For any conditionally guided model, FID varies strongly and non-monotonically with guidance strength while prompt alignment moves the other way. A single FID for such a model is a choice of operating point, and reporting it without the scale is closer to a selection than a measurement.
12 flashcards for this concept
Click a card to reveal the answer.