Generative Adversarial Networks

Likelihood-based generative models of images produced blurry samples because they optimise a per-pixel loss and blur is the optimal hedge under uncertainty. The 2014 proposal was to stop specifying a loss at all and learn one: train a generator to produce images and a discriminator to tell real from generated, and let the discriminator's gradient be the generator's training signal (Goodfellow et al., 2014, Generative Adversarial Nets, arXiv:1406.2661). No explicit density, no bound, no reconstruction term.

The game

Generator \(G\) maps noise \(z \sim p(z)\) to samples; discriminator \(D\) outputs the probability that its input is real. They optimise opposing objectives:

For a fixed \(G\), the optimal discriminator is \(D^*(x) = p_{\text{data}}(x) / (p_{\text{data}}(x) + p_g(x))\). Substituting it back shows the generator is minimising the Jensen-Shannon divergence between the data and model distributions, up to a constant. That is an elegant result with a nasty corollary: when the two distributions have disjoint support, which is exactly the situation early in training with high-dimensional image manifolds, JS divergence is constant and its gradient is zero. The generator gets no signal.

In practice the fix used from day one is the non-saturating loss, where the generator maximises \(\log D(G(z))\) instead of minimising \(\log(1 - D(G(z)))\), giving strong gradients when samples are obviously fake. Wasserstein GAN went further, replacing JS with an earth-mover distance that stays informative under disjoint support, at the price of needing a constraint (weight clipping, later a gradient penalty) on the critic (Arjovsky et al., 2017, arXiv:1701.07875).

Mode collapse

The generator's objective rewards fooling the current discriminator, not covering the data. If one narrow family of outputs fools \(D\), producing only that family is a local optimum, and the model quietly stops representing whole regions of the data distribution. A face generator that emits only front-facing portraits has collapsed even if every sample is photorealistic.

This is structural. A likelihood-trained model pays an unbounded penalty for assigning near-zero probability to observed data; an adversarial model pays nothing at all for ignoring a mode the discriminator has stopped policing. Minibatch discrimination, unrolled optimisation, and multiple discriminators all mitigate it, and none removes it.

What GANs are still good at

The lasting wins are architectural and evaluative rather than conceptual. DCGAN established the convolutional generator and discriminator conventions that survived into other model families (Radford et al., 2015, arXiv:1511.06434). StyleGAN's mapping network and per-layer style modulation produced the most controllable and highest-fidelity face models of their era and remain a reference for latent-space editing (Karras et al., 2018, arXiv:1812.04948). Fréchet Inception Distance, introduced to compare GANs, became the default image-generation metric for every family that followed (Heusel et al., 2017, arXiv:1706.08500).

The changeover is dated precisely: classifier-guided diffusion beat BigGAN-deep on ImageNet sample quality, reporting FID 2.97 at 128×128 and 4.59 at 256×256, with better distribution coverage (Dhariwal and Nichol, 2021, Diffusion Models Beat GANs on Image Synthesis, arXiv:2105.05233). GANs kept a niche where single-step sampling matters: real-time super-resolution, on-device enhancement, and, increasingly, as the adversarial loss inside distillation schemes that compress many-step diffusion samplers into one or four steps.

When it breaks

• Training is a two-player game, so there is no loss curve to read. A falling generator loss can mean the discriminator is losing, not that samples improved. You evaluate by sampling and by FID, not by the objective.
• Balance is fragile. A discriminator that gets too good returns vanishing gradients; too weak and it provides no signal. The stable region depends on learning rates, architecture, and dataset, and it moves during training.
• No likelihood, so no anomaly scoring. You cannot ask a GAN how probable a given input is, which rules it out for density estimation, compression, and outlier detection.
• Evaluation metrics are gameable. FID is sensitive to the feature extractor and to sample count, and rewards matching Inception statistics, which is not the same as matching the data distribution.