Mode Collapse and Training Instability

Train a GAN on MNIST and watch it produce beautiful, varied sevens. Only sevens. The discriminator cannot fault any individual sample, the generator loss is low, and nine tenths of the data distribution has been abandoned. Nothing has gone wrong mechanically. The generator found a strategy the objective rewards.

The missing incentive

A maximum-likelihood model pays directly for failing to cover data: any real point assigned near-zero probability contributes an unbounded penalty to the loss. A GAN generator pays no such price. Its objective is stated entirely in terms of the samples it produces, and it is judged one sample at a time by a discriminator that also sees only one sample at a time. Producing a narrow, high-quality distribution is a legitimate way to make every individual sample look real.

This is the mode-seeking versus mode-covering distinction, and it is the reason GAN samples were sharper than VAE samples for years while covering less of the distribution. Reverse-KL-like objectives concentrate on modes; forward-KL-like objectives spread to cover. Neither is universally correct, and which failure you prefer depends on whether an unrealistic sample or a missing mode is worse for the application.

Full collapse, partial collapse, and cycling

Full collapse is the textbook case: the generator maps every \(z\) to nearly the same output. It is rare in well-tuned systems and easy to spot.

Partial collapse is the common case and much harder to detect. The generator covers most of the distribution but drops specific regions: a demographic group, a rare object class, an unusual lighting condition. Aggregate metrics barely move, because FID is dominated by the bulk of the distribution, and the missing region is only visible if you go looking for it. This is where the fairness and coverage consequences of generative models actually live.

Cycling collapse is the dynamical version. The generator collapses onto one mode, the discriminator learns to reject that mode, the generator jumps to another, and the process repeats without ever settling. Sample grids taken at different checkpoints look different and each looks impoverished. It is a signature of the two-player dynamics failing to converge rather than of a bad generator.

What the mitigations change

Minibatch discrimination and minibatch standard deviation give the discriminator access to statistics across a batch rather than only single samples. A collapsed generator produces a batch with unnaturally low variance, which a discriminator with batch-level features can detect and penalise. This directly repairs the missing incentive: coverage becomes something the discriminator can see. The minibatch standard deviation layer used in the StyleGAN family is the cheap and widely adopted version.

Unrolled GANs let the generator optimise against a discriminator that has been updated a few steps into the future, so it cannot exploit a myopic discriminator response. Effective, and expensive enough that it stayed a research technique.

Wasserstein and hinge losses change the divergence so that gradients remain informative when the generator's distribution is far from the data, attacking the vanishing-gradient half of the problem rather than the incentive half.

Spectral normalisation constrains the discriminator's Lipschitz constant by dividing each weight matrix by its largest singular value. It is cheap, requires no extra loss term, and became the default stabiliser in much of the literature because it removes the discriminator's ability to produce arbitrarily sharp decision boundaries and therefore arbitrarily large gradients.

When it breaks

FID hides partial collapse. FID compares Gaussian fits to Inception features of two sample sets, and dropping a small mode moves those first two moments very little. Precision and recall metrics for generative models, which separate sample quality from distribution coverage, are the tools that make the failure visible, and reporting only FID is a choice that conceals it.

Mitigations trade against sharpness. Anything that forces coverage moves the model toward mode-covering behaviour, and mode-covering models produce more low-quality samples in the low-density regions between modes. Teams that push coverage hard often find their sample quality metric worsening, and that tradeoff is real rather than a tuning failure.

Collapse can be inherited. Fine-tuning a pretrained generator on a narrow dataset, or distilling one generator into another, propagates and usually amplifies whatever coverage gaps the parent had. There is no mechanism in the adversarial objective that would recover a mode the teacher never produced.

Diffusion models mostly do not have this problem, and pay elsewhere. Because they are trained with a maximum-likelihood-style denoising objective over the full data distribution, they cover well by construction. Their failure modes are slow sampling and blurry low-density outputs, which is the other side of the same tradeoff rather than a free improvement.