Why GANs Lost Image Generation, and Why Adversarial Losses Are Everywhere Anyway

On ImageNet at 256×256, BigGAN-deep scored a precision of 0.87 and a recall of 0.28. The guided diffusion model that replaced it scored 0.82 and 0.52 (Dhariwal and Nichol, 2021, Diffusion Models Beat GANs on Image Synthesis, arXiv:2105.05233). The GAN's samples were, if anything, more often inside the manifold of real images. What it could not do was reach most of that manifold: nearly three quarters of real images had no generated neighbour. Diffusion gave up five points of precision, bought twenty-four points of recall, and FID fell from 6.95 to 4.59.

That row explains why GANs lost image generation, and, read backwards, why adversarial losses stayed. When something else guarantees coverage (an image to reconstruct, a spectrogram to invert, a diffusion teacher that already learned the distribution), what a discriminator does badly stops mattering, and what it does well, telling a sharp real sample from a plausible blurry one, becomes the best tool available. As of September 2026 the latent autoencoders in the Stable Diffusion lineage, several of the fastest distilled image and video generators, and the neural codecs that tokenise speech all train against a discriminator.

Why this matters: "GANs are dead" is the wrong lesson and "GANs are back" is the wrong correction. The useful lesson is a design rule: an adversarial loss is a fidelity term, not a coverage term. Knowing why the minimax objective drops modes and saturates, and why conditioning neutralises both, tells you where a discriminator pays and where it will quietly collapse a model.

TL;DR

• With an optimal discriminator the GAN objective equals \(2\,\mathrm{JSD}(p_{\text{data}} \| p_g) - \log 4\), a divergence capped at \(\log 2\) and constant when supports do not overlap, so a perfect discriminator gives no gradient.
• Dropping a mode costs a GAN a bounded amount (0.216 nats in the two-mode example below) and a maximum-likelihood model an infinite amount. Mode dropping is priced in.
• Guided diffusion reached FID 4.59 on ImageNet 256 against BigGAN-deep's 6.95, recall 0.52 against 0.28, but used about 962 V100-days against an estimated 128 to 256, and 250 network passes per sample instead of one.
• Naive scaling failed: widening StyleGAN2 5.7× worsened FID from 29.91 to 34.07, and BigGAN's discriminator hit over 98% training accuracy and chance on validation before collapsing.
• GANs still win on speed: GigaGAN made a 512px image in 0.13 s at COCO FID 9.09, and R3GAN reached ImageNet-64 FID 2.09 in one step against EDM's 2.23 at 79.
• Adversarial losses survive wherever conditioning supplies coverage: VQGAN and latent-diffusion autoencoders, HiFi-GAN (MOS 4.36 against 4.45 for real speech), EnCodec, DAC, and adversarial diffusion distillation.

At a Glance

flowchart LR
    subgraph Game["The adversarial game"]
        Z["Noise z"] --> G["Generator G"]
        G --> FAKE["Generated sample"]
        REAL["Real sample"] --> D["Discriminator D"]
        FAKE --> D
    end
    D --> FID["Fidelity signal: sharp, realistic"]
    D --> GAP["Coverage gap: unseen modes cost little"]
    GAP --> LOST["Lost: unconditional generation"]
    FID --> WON["Won: loss beside a coverage anchor"]
    WON --> USES["Autoencoders, vocoders, codecs, distillation"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    class Z,REAL blue
    class G,D purple
    class FAKE teal
    class FID,WON,USES emerald
    class GAP,LOST rose

One discriminator yields a strong realism signal and a weak, bounded penalty for missing modes. Which dominates depends on whether anything else anchors coverage.

Seven Years at the Top

Before 2014, deep generative image models were likelihood-based and blurry, because under a per-pixel Gaussian the optimal hedge is the average. Goodfellow and colleagues skipped the explicit density: a generator maps noise to samples, a discriminator classifies real against generated, and the discriminator's gradient teaches the generator (Goodfellow et al., 2014, arXiv:1406.2661). The loss function itself was learned.

The next five years were fixes to named failures. DCGAN found convolutional architectures that trained at all (Radford, Metz and Chintala, 2015, arXiv:1511.06434). Arjovsky and Bottou proved why training was fragile: distributions on low-dimensional manifolds almost never overlap, so a perfect discriminator exists and gradients vanish (Arjovsky and Bottou, 2017, arXiv:1701.04862). Wasserstein GAN replaced the divergence (Arjovsky, Chintala and Bottou, 2017, arXiv:1701.07875), WGAN-GP replaced weight clipping with a gradient penalty (Gulrajani et al., 2017, arXiv:1704.00028), and spectral normalisation bounded the discriminator's Lipschitz constant cheaply (Miyato et al., 2018, arXiv:1802.05957).

Then scale. BigGAN trained on 128 to 512 TPUv3 cores and lifted ImageNet 128×128 Inception Score from 52.52 to 166.5 and FID from 18.6 to 7.4 (Brock, Donahue and Simonyan, 2018, arXiv:1809.11096). StyleGAN's mapping network and per-layer style modulation produced the most editable latent space the field had seen (Karras, Laine and Aila, 2018, arXiv:1812.04948).

timeline
    title From minimax game to loss component
    2014 : Goodfellow et al. propose Generative Adversarial Nets
    2015 : DCGAN makes convolutional GANs trainable
    2017 : Wasserstein GAN and WGAN-GP attack vanishing gradients
         : FID introduced alongside the two time-scale update rule
    2018 : Spectral normalisation bounds the discriminator
         : BigGAN scales to ImageNet, StyleGAN introduces style modulation
    2019 : Kynkaanniemi et al. split quality into precision and recall
    2020 : HiFi-GAN vocoder nears human MOS
         : VQGAN puts a patch discriminator inside an autoencoder
    2021 : Dhariwal and Nichol show diffusion beats GANs on ImageNet
    2023 : GigaGAN scales a GAN to 1B parameters for text-to-image
         : Adversarial Diffusion Distillation powers SDXL-Turbo
    2024 : R3GAN argues GAN training need not be fragile
         : Mimi codec trains with adversarial losses only
    2025 : Seaweed-APT generates 720p video in one step

Two evaluation papers matter as much as any model. The Fréchet Inception Distance compressed fidelity and diversity into one number (Heusel et al., 2017, arXiv:1706.08500). Kynkäänniemi et al. pulled them apart again with a k-nearest-neighbour estimate of precision and recall (Kynkäänniemi et al., 2019, arXiv:1904.06991). Without that second paper, 2021 reads as "diffusion got a better score". With it, it reads as "diffusion covered the distribution and GANs did not".

[IMAGE: Lineage chart with rows for Losses, Regularisers, Architectures, Metrics and Successors on a 2014 to 2025 axis; arrows show spectral normalisation feeding BigGAN, StyleGAN2 feeding GigaGAN and R3GAN, and the PatchGAN discriminator feeding VQGAN and latent-diffusion autoencoders. Caption: "The GAN literature is a chain of fixes for named failures; the right-hand branches transplanted the discriminator rather than abandoning it."]

How the Adversarial Objective Actually Works

The optimal discriminator and the divergence it implies

The original game is

Pointwise, the discriminator picks \(y = D(x)\) to maximise \(a \log y + b \log(1-y)\) with \(a = p_{\text{data}}(x)\), \(b = p_g(x)\). Setting \(a/y - b/(1-y) = 0\) gives

the Bayes posterior that a sample is real under equal priors. Substituting back, with \(m = \tfrac{1}{2}(p_{\text{data}} + p_g)\),

Facing a perfect discriminator, the generator minimises Jensen-Shannon divergence, and at \(p_g = p_{\text{data}}\) the discriminator outputs \(1/2\) everywhere (Goodfellow et al., 2014). JSD is symmetric, bounded by \(\log 2\), and depends only on overlap, not on distance between points. All three properties come back to bite.

Why the bound becomes a gradient problem

If the supports are disjoint, \(D^*\) is 1 on data and 0 on samples and JSD is \(\log 2\) however far apart they sit. A generator one pixel off the manifold and one producing static score the same. Arjovsky and Bottou showed that for manifold-supported distributions in high dimensions this is the generic case (Arjovsky and Bottou, 2017).

The 2014 paper's practical fix was the non-saturating loss: maximise \(\log D(G(z))\) instead of minimising \(\log(1 - D(G(z)))\). With \(D = \sigma(\ell)\) the logit gradients are

When the discriminator confidently rejects a sample, \(\sigma(\ell) \approx 0\): the first gradient vanishes, the second is near its maximum. But the trick changes the objective. Arjovsky and Bottou proved that with an optimal discriminator

The reverse KL explodes when the generator puts mass where data has none (a fake-looking sample) and is indifferent when it puts no mass where data has plenty (a missing mode). The minus-JSD term even rewards pushing distributions apart. So the objective most practical GANs optimised is harsh on implausible samples and lenient on absent ones. High precision with low recall is not a BigGAN hyperparameter accident; it is the signature of the loss.

Mode dropping from the generator's side

A second reason applies even where the divergence would object. The generator is a pushforward: its gradient is \(\nabla_x D(x)\) evaluated at generated points and back-propagated through \(G\). If no generated sample lands near a data mode, no term in the generator's loss mentions it, and the discriminator's knowledge that the region is "real" never reaches \(G\). Maximum likelihood evaluates its loss at the data points instead, so a model assigning near-zero density to a training image pays \(-\log p_\theta(x) \to \infty\) on it every epoch.

Constraints: Wasserstein, spectral norm, zero-centred penalties

By Kantorovich-Rubinstein duality,

which shrinks smoothly as disjoint supports approach. The difficulty moved into enforcing the 1-Lipschitz constraint: clipping works badly, a gradient penalty on interpolates works better (Gulrajani et al., 2017), and spectral normalisation divides each weight matrix by its estimated top singular value (Miyato et al., 2018). Theory then complicated the picture: unregularised GANs, and WGAN-GP with finitely many critic steps, are not always locally convergent, while zero-centred penalties such as \(R_1 = \tfrac{\gamma}{2}\,\mathbb{E}_{p_{\text{data}}}\|\nabla_x D(x)\|^2\) are (Mescheder, Geiger and Nowozin, 2018, arXiv:1801.04406). \(R_1\) became the StyleGAN default and sits inside ADD's discriminator loss today.

What scale did to the discriminator

BigGAN's authors instrumented their own failure. Generator spectral norms grew through training and exploded at collapse. A discriminator gradient penalty with \(\gamma = 10\) stabilised training and cut Inception Score by 45%; \(\gamma = 1\), the weakest value that prevented sudden collapse, still cost 20%. The uncollapsed discriminator classified ImageNet training images with over 98% accuracy and validation images at 50 to 55%, chance (Brock et al., 2018). It was memorising. The best results came from letting collapse happen late and early-stopping.

A memorising discriminator is exactly where JSD's weaknesses bite: more capacity separates finite training data from everything else more easily, with less useful gradients off the data. Diffusion's denoising objective has no second network to overfit and no equilibrium to lose. It is regression against a fixed target, and it scales like regression.

BigGAN's truncation trick and diffusion's classifier guidance are the same dial, trading variety for fidelity. Diffusion starts at high recall and spends it; a GAN starts with little to spend.

Precision and recall as manifolds

Kynkäänniemi et al. embed images with VGG-16 and draw around each feature vector a hypersphere reaching its \(k\)-th nearest neighbour within its own set (\(k = 3\)). Then

Precision asks how many generated samples look like something real; recall asks how much real data has a generated neighbour. A collapsed generator aces the first and fails the second, and FID blends both into a number that cannot say which you have.

[IMAGE: Two-panel plot of BigGAN training. Left: top singular value of three generator layers rising slowly then spiking at collapse. Right: discriminator accuracy on ImageNet train climbing above 98% while validation stays near 50 to 55%. Caption: "BigGAN's discriminator learns the training set, not the distribution; collapse follows the memorisation."]

Seeing It in Motion

Where exactly does the gradient stop reaching the generator? One training step shows it.

sequenceDiagram
    participant Data as Data loader
    participant G as Generator
    participant D as Discriminator
    participant Optim as Optimisers
    Data->>D: Real minibatch
    G->>D: Generated minibatch, detached
    D->>Optim: Real vs fake loss plus R1 penalty
    Optim->>D: Update discriminator
    G->>D: Fresh generated minibatch
    D->>G: Gradient of D at generated points only
    G->>Optim: Non-saturating generator loss
    Optim->>G: Update generator
    Note over G,D: Modes with no generated samples never enter this gradient

Why is the same discriminator safe elsewhere? Because every surviving design pairs it with an anchor.

flowchart TB
    subgraph AE["Autoencoder: VQGAN, LDM"]
        A1["Input image"] --> A2["Encoder and decoder"]
        A2 --> A3["Reconstruction loss anchors content"]
        A2 --> A4["Patch discriminator adds sharpness"]
    end
    subgraph VOC["Vocoder and codec: HiFi-GAN, DAC"]
        V1["Mel frames or tokens"] --> V2["Waveform generator"]
        V2 --> V3["Spectral loss anchors content"]
        V2 --> V4["Period and STFT discriminators"]
    end
    subgraph DIS["Distillation: ADD, DMD2, APT"]
        S1["Diffusion teacher"] --> S2["One-step student"]
        S1 --> S3["Score loss anchors coverage"]
        S2 --> S4["Feature discriminator adds detail"]
    end

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class A1,V1,S1 blue
    class A2,V2,S2 purple
    class A3,V3,S3 emerald
    class A4,V4,S4 amber

The anchor pins each output to a target, so the generator cannot ignore a mode the input already names. Overlap is high from step one and JSD's pathologies have little room. Unanchored runs follow a lifecycle practitioners learn the hard way.

stateDiagram-v2
    [*] --> Warmup
    Warmup --> Balanced: D and G losses oscillate
    Balanced --> DOverfits: D memorises training set
    Balanced --> ModeShrink: G finds samples D accepts
    DOverfits --> Collapse: Gradients explode
    ModeShrink --> Collapse: Diversity lost
    Balanced --> EarlyStop: Best FID checkpoint kept
    Collapse --> EarlyStop: Roll back
    EarlyStop --> [*]

[IMAGE: BigGAN-style samples for one ImageNet class at truncation 2.0, 1.0 and 0.5. Caption: "Truncation trades recall for precision; at 0.5 every dog looks excellent and alike."]

By the Numbers

Sources: BigGAN (Brock et al., 2018); ImageNet comparisons and compute, Tables 5 and 10 (Dhariwal and Nichol, 2021), where BigGAN-deep compute is the authors' conversion of TPU estimates and is approximate; StyleGAN2 ablation and text-to-image comparison, Tables 1 and 2 (Kang et al., 2023, arXiv:2303.05511), with rival figures partly quoted from papers using different data; R3GAN, Tables 4 and 8 (Huang et al., 2024, arXiv:2501.05441); HiFi-GAN, Table 1 (Kong, Kim and Bae, 2020, arXiv:2010.05646); ADD, Table 2, A100 timings (Sauer et al., 2023, arXiv:2311.17042); DMD2 (Yin et al., 2024, arXiv:2405.14867); Mimi, Table 4 (Défossez et al., 2024, arXiv:2410.00037).

Diffusion's win was not free: an ADM-G checkpoint at 393 V100-days reached only FID 6.49, level with BigGAN-deep. Nor was it a compute artifact: at every resolution, diffusion's recall sits far above the GAN's. Compute bought the FID. The objective bought the coverage.

[IMAGE: Scatter of ImageNet 256 results, precision on x and recall on y: BigGAN-deep (0.87, 0.28), ADM (0.69, 0.63), ADM-G 25 steps (0.81, 0.49), ADM-G 250 steps (0.82, 0.52); marker size proportional to network passes per sample. Caption: "The GAN wins the x-axis; diffusion wins the y-axis and pays in network evaluations."]

A Concrete Example

Everything above fits on paper with three outcomes \(\{a, b, c\}\), natural logarithms, and

Step 1: optimal discriminator.

Step 2: value function. Real term: \(0.5\log 0.7143 + 0.3\log 0.6 + 0.2\log 0.25 = -0.1682 - 0.1532 - 0.2773 = -0.5987\). Generated term: \(0.2\log 0.2857 + 0.2\log 0.4 + 0.6\log 0.75 = -0.2506 - 0.1833 - 0.1726 = -0.6064\). So \(V(D^*, G) = -1.2052\).

Step 3: recover JSD two ways. From the theorem, \(\mathrm{JSD} = (V + \log 4)/2 = (-1.2052 + 1.3863)/2 = 0.0906\) nats. Directly, \(m = (0.35, 0.25, 0.40)\); \(\mathrm{KL}(p_{\text{data}} \| m) = 0.1783 + 0.0547 - 0.1386 = 0.0944\) and \(\mathrm{KL}(p_g \| m) = -0.1119 - 0.0446 + 0.2433 = 0.0867\); their mean is \(0.0906\). The routes agree.

Step 4: drop a mode. Two outcomes, \(p_{\text{data}} = (0.5, 0.5)\), collapsed \(p_g = (1, 0)\). Then \(D^* = (1/3, 1)\), \(m = (0.75, 0.25)\), and \(\mathrm{JSD} = \tfrac{1}{2}(0.1438 + 0.2877) = 0.2158\), under the \(\log 2 = 0.6931\) ceiling. Maximum likelihood scores this generator \(\mathrm{KL}(p_{\text{data}} \| p_g) = \infty\). The reverse KL that drives the non-saturating gradient is \(0.6931\): finite, the price of a perfectly plausible distribution missing half the world.

Step 5: disjoint supports. Put outcomes on a line. \(p_{\text{data}}\) uniform on \(\{0, 1\}\), \(p_g\) uniform on \(\{2, 3\}\): JSD is \(0.6931\). Move \(p_g\) to \(\{4, 5\}\): still \(0.6931\), with no hint of direction. Wasserstein-1 is 2, then 4. That difference is WGAN's motivation.

Step 6: the gradient received. A sample rejected with \(D = 0.01\) has logit \(\ell = \log(0.01/0.99) = -4.595\). The saturating loss yields logit gradient \(-0.01\); the non-saturating loss yields \(-0.99\), 99 times stronger.

Step 7: read the collapse. One dimension, \(k = 1\). Real features \(\{0, 1, 2, 10, 11, 12\}\) each have radius 1. A collapsed generator emits \(\{0.5, 1.2, 1.8\}\) with radii 0.7, 0.6, 0.6. Each generated point is within 1 of a real point: precision \(3/3 = 1.0\). Real points 0, 1, 2 fall inside generated spheres (distances 0.5, 0.2, 0.2); 10, 11, 12 do not: recall \(3/6 = 0.5\). One cluster learned perfectly, one not at all, reported in two numbers where FID gives one.

[IMAGE: Three-row figure. Row 1: paired bars of p_data and p_g over a, b, c with D* above each. Row 2: two-mode collapse with mixture m dashed, JSD 0.216 against forward KL marked infinite. Row 3: number line with real points 0 to 2 and 10 to 12, generated points near 1 with radius circles, precision 1.0 and recall 0.5. Caption: "Optimal discriminator, JSD, a dropped mode and precision-recall on a napkin."]

Where It Breaks

Losses that do not read as progress

A falling generator loss can mean better samples or a weaker discriminator. In BigGAN the discriminator loss sat near zero and jumped at collapse (Brock et al., 2018). Monitor held-out discriminator accuracy, spectral norms and precision-recall instead, with checkpoints to roll back to.

Metrics that can be gamed, in both directions

The "GANs lost" verdict rests on FID, so its flaws are part of the argument. Matching the top-N ImageNet class histogram of real images lowers FID substantially without improving images, and an ImageNet-pretrained FastGAN matched StyleGAN2's FID while losing in human evaluation (Kynkäänniemi et al., 2022, arXiv:2203.06026). R3GAN's tables accordingly flag pretrained-feature GANs such as StyleGAN-XL for "ImageNet feature leakage". The bias runs the other way too: the largest human study of generative models found no metric correlating strongly with perceived realism and FID under-rewarding diffusion, partly through over-reliance on Inception-V3 (Stein et al., 2023, arXiv:2306.04675).

The field disagrees on whether GANs are inherently hard

Lucic et al. found that most GAN variants reach similar scores with enough tuning and restarts, and that none consistently beat the original non-saturating GAN (Lucic et al., 2018, arXiv:1711.10337). Huang et al. argue the "hard to train" claim is itself wrong: a relativistic loss with both \(R_1\) and \(R_2\) penalties is locally convergent, lets you strip StyleGAN2's tricks, and beats it (Huang et al., 2024). Both can hold: the loss menu mattered little, regularisation mattered a lot. Neither shows a GAN matching diffusion's recall on open-vocabulary web data.

Scaling the naive way makes things worse

Widening StyleGAN2 5.7× raised FID-10k from 29.91 to 34.07. GigaGAN's gains came from attention, adaptive convolution kernels, matching-aware discriminators, multi-scale training and a CLIP loss, each ablated, and its authors still show failures against DALL·E 2 on photorealism and composition (Kang et al., 2023). Where diffusion absorbs parameters along a predictable loss curve, a GAN needs architecture work per order of magnitude.

Discriminators inside hybrids fail too

VQGAN weights its adversarial term adaptively, \(\lambda = \nabla_{G_L}[\mathcal{L}_{\text{rec}}] / (\nabla_{G_L}[\mathcal{L}_{\text{GAN}}] + \delta)\), a ratio of gradient norms at the decoder's last layer, and still recommends \(\lambda = 0\) for at least one epoch, because a discriminator switched on against an untrained decoder dominates it (Esser, Rombach and Ommer, 2021, arXiv:2012.09841). EnCodec added a loss balancer so each weight means a fraction of total gradient, then gave adversarial loss weight 3 against 0.1 for time-domain reconstruction (Défossez et al., 2022, arXiv:2210.13438). LADD was motivated by how expensive and difficult ADD's fixed DINOv2 discriminator was to optimise (Sauer et al., 2024, arXiv:2403.12015). And one-step students inherit narrowness: ADD reports students adopting their teacher's traits, and SDXL-Turbo's model card lists fixed 512×512 output, illegible text and unreliable faces (Stability AI, SDXL-Turbo model card).

Alternative Designs

The real choice is which term supplies coverage and which supplies realism. Pure GANs remain right for narrow domains where latency or editability beats breadth.

How It Is Used in Practice

Latent image models. Latent diffusion trains its autoencoder with a perceptual loss and a patch-based adversarial objective so reconstructions avoid pixel-loss blur (Rombach et al., 2021, arXiv:2112.10752), a recipe inherited from VQGAN's patch discriminator. Nobody calls the result a GAN.

Fast image and video generation. SDXL-Turbo is ADD applied to SDXL 1.0: a score-distillation loss from the frozen teacher plus a hinge loss from a DINOv2-feature discriminator, sampling in one to four steps (Sauer et al., 2023); in Stability AI's own user study, four student steps beat the 50-step teacher in most comparisons, a vendor result. SD3-Turbo applies the latent variant to an 8B model; DMD2 adds a GAN loss on real images so the student can beat its teacher; and Seaweed-APT post-trains a diffusion model adversarially against real data to produce two-second 1280×720, 24fps video in one forward pass, in real time (Lin et al., 2025, arXiv:2501.08316).

Speech and audio. HiFi-GAN observed that speech is a sum of periodic signals, so its multi-period discriminator inspects waveforms at several periods, paired with a mel-spectrogram L1 loss (Kong, Kim and Bae, 2020). Codecs took over the recipe: SoundStream trains with adversarial and reconstruction losses (Zeghidour et al., 2021, arXiv:2107.03312); DAC compresses 44.1 kHz audio about 90× at 8 kbps using the multi-period discriminator, a multi-band STFT discriminator, hinge loss and feature matching (Kumar et al., 2023, arXiv:2306.06546); Moshi's Mimi keeps only feature-matching and discriminator losses.

Operations. The discriminator is discarded at inference, so its cost lands in training: two networks in memory, extra hyperparameters, and human listening or viewing tests, because objective metrics disagree with people exactly where the adversarial term works.

[IMAGE: Spoken-dialogue stack: language model emitting codec tokens, codec decoder emitting waveform, with training-only multi-period and STFT discriminators in a dashed box attached to the decoder and crossed out on the inference path. Caption: "The discriminator exists only during training; at inference the decoder is a plain feed-forward network."]

Insights Worth Remembering

1. GANs lost on recall, not fidelity. BigGAN-deep out-precisioned the diffusion model that beat it; its recall was barely half.

2. The non-saturating trick changed the divergence. It cured vanishing gradients by optimising reverse KL minus twice JSD, harsh on fake samples and lenient on absent ones. The fix that made GANs trainable baked in mode dropping.

3. Conditioning makes a discriminator safe. When the input names the wanted output, the generator cannot drop a mode and the discriminator becomes a learned perceptual loss.

4. Learned losses can beat hand-written ones where humans judge. Mimi's adversarial-only codec lost on VisQOL and won MUSHRA by 22 points.

5. Diffusion's win was expensive, and distillation is the refund. Roughly four to seven times the training compute and 250 times the inference passes; adversarial distillation pays the inference bill back.

6. FID cannot referee this fight alone. It rewards ImageNet class statistics and under-rewards diffusion realism; report precision, recall and human studies.

Open Questions

Can a GAN reach diffusion-level recall on open-vocabulary data? Measured: R3GAN beats diffusion baselines on ImageNet-64 and FFHQ FID in one step; GigaGAN matched Stable Diffusion's COCO FID. Unmeasured: precision and recall at web scale. Whether the limit is the objective or unfunded engineering is open.

How much diversity do adversarially distilled students lose? Shown: ADD students inherit their teacher's diversity profile. Unknown: per-prompt diversity loss at scale, which standard metrics were not designed to measure.

Why does adversarial-only training help codecs? Measured: Mimi's MUSHRA rose from 58.8 to 81.0. Plausible but unproven: reconstruction losses spend decoder capacity on phase detail listeners do not perceive. Transfer to image autoencoders, where downstream models may need faithful reconstructions, is untested.

What should replace FID across families? Evidence: Inception features leak class information; DINOv2 features evaluate more richly. Open: separating memorisation, mode shrinkage and underfitting, which Stein et al. report no current metric does.

Will adversarial post-training replace distillation? Shown: Seaweed-APT trains a one-step video generator against real data. Speculative: that diffusion pre-training plus adversarial post-training becomes the default.

Sources and Further Reading

1. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). "Generative Adversarial Nets." NeurIPS 2014. arXiv:1406.2661
2. Arjovsky, M., & Bottou, L. (2017). "Towards Principled Methods for Training Generative Adversarial Networks." ICLR 2017. arXiv:1701.04862
3. Arjovsky, M., Chintala, S., & Bottou, L. (2017). "Wasserstein Generative Adversarial Networks." ICML 2017. arXiv:1701.07875
4. Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). "Spectral Normalization for Generative Adversarial Networks." ICLR 2018. arXiv:1802.05957
5. Mescheder, L., Geiger, A., & Nowozin, S. (2018). "Which Training Methods for GANs do actually Converge?" ICML 2018. arXiv:1801.04406
6. Brock, A., Donahue, J., & Simonyan, K. (2019). "Large Scale GAN Training for High Fidelity Natural Image Synthesis." ICLR 2019. arXiv:1809.11096
7. Karras, T., Laine, S., & Aila, T. (2019). "A Style-Based Generator Architecture for Generative Adversarial Networks." CVPR 2019. arXiv:1812.04948
8. Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). "Improved Precision and Recall Metric for Assessing Generative Models." NeurIPS 2019. arXiv:1904.06991
9. Dhariwal, P., & Nichol, A. (2021). "Diffusion Models Beat GANs on Image Synthesis." NeurIPS 2021. arXiv:2105.05233
10. Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023). "Scaling up GANs for Text-to-Image Synthesis." CVPR 2023. arXiv:2303.05511
11. Huang, Y., Gokaslan, A., Kuleshov, V., & Tompkin, J. (2024). "The GAN is dead; long live the GAN! A Modern GAN Baseline." NeurIPS 2024. arXiv:2501.05441
12. Lucic, M., Kurach, K., Michalski, M., Gelly, S., & Bousquet, O. (2018). "Are GANs Created Equal? A Large-Scale Study." NeurIPS 2018. arXiv:1711.10337
13. Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., & Lehtinen, J. (2023). "The Role of ImageNet Classes in Fréchet Inception Distance." ICLR 2023. arXiv:2203.06026
14. Stein, G., Cresswell, J. C., Hosseinzadeh, R., et al. (2023). "Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models." NeurIPS 2023. arXiv:2306.04675
15. Esser, P., Rombach, R., & Ommer, B. (2021). "Taming Transformers for High-Resolution Image Synthesis." CVPR 2021. arXiv:2012.09841
16. Kong, J., Kim, J., & Bae, J. (2020). "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis." NeurIPS 2020. arXiv:2010.05646
17. Sauer, A., Lorenz, D., Blattmann, A., & Rombach, R. (2023). "Adversarial Diffusion Distillation." arXiv:2311.17042
18. Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., & Zeghidour, N. (2024). "Moshi: a speech-text foundation model for real-time dialogue." arXiv:2410.00037

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.