Adversarial Losses as a Component

Diffusion displaced GANs as the way to generate images, and the adversarial loss did not go away. It sits inside the autoencoder of every latent diffusion model, inside the neural vocoders that turn spectrograms into waveforms, and inside most super-resolution and image-restoration systems. The generative model lost; the loss function won a different job.

The problem it solves

Train any reconstruction model with L2 loss and the output is blurry. The reason is not a limitation of the network. Given inherent ambiguity about fine detail, the L2-optimal prediction is the conditional mean over all plausible completions, and the average of many sharp textures is a smooth one. The model is not failing; it is being asked for an average and delivering one.

Perceptual losses, comparing features from a pretrained network rather than pixels, help by measuring similarity in a space where averaging is less destructive. They do not fully solve it, because they still specify a single target to move toward.

An adversarial loss asks a categorically different question. Rather than "how close is this output to the target", it asks "does this output look like it came from the real distribution". A blurry patch is immediately identifiable as unreal regardless of how close its mean is, so the discriminator penalises blur directly. That is the property no regression loss has.

The patch discriminator

The discriminator used for this job is usually a PatchGAN: a small fully convolutional network that classifies overlapping local patches, typically 70x70 pixels, rather than the whole image, and averages the results. Introduced with pix2pix (Isola et al., 2017, arXiv:1611.07004), it fits the task precisely. Global structure is already pinned down by the reconstruction loss, so the discriminator's only job is texture realism, which is a local property. A patch discriminator has far fewer parameters, is more stable to train, and generalises across image sizes because it is fully convolutional.

The combined objective is a weighted sum: a reconstruction term for fidelity to the specific input, a perceptual term for mid-level structure, and an adversarial term for texture. The adversarial weight is small, often one or two orders of magnitude below the others. It is a finishing pass, not the main signal.

Why this arrangement is stable when standalone GANs are not

Most GAN instability comes from the generator being free to move its entire output distribution anywhere. Here it is not free: the reconstruction loss pins each output near a specific target, so the generator cannot collapse to a single mode or wander into a region where the discriminator gives no gradient. Mode collapse is structurally impossible when every output is anchored to its own input. The adversarial term operates in the narrow space the reconstruction loss leaves open, which is precisely where it is well behaved.

This is why the latent diffusion autoencoder, HiFi-GAN and similar vocoders, and ESRGAN-style super-resolution all use adversarial losses without the elaborate stabilisation machinery a standalone GAN needs.

When it breaks

Adversarial weight is a fidelity-realism dial with no free setting. Raise it and textures sharpen while the output drifts further from the true input, inventing plausible detail that was not there. In photography that is often desirable; in medical or scientific imaging it is a serious problem, because hallucinated texture is indistinguishable from measured texture in the final image.

Perceptual and adversarial losses can agree on something wrong. Both are trained on natural image statistics, so both reward output that looks like a natural image. On data that is not natural imagery, they can jointly push toward a photographic appearance the domain does not have.

Checkerboard artefacts have a specific cause. Transposed convolutions with kernel size not divisible by stride produce uneven overlap, and adversarial training amplifies the resulting periodic pattern rather than smoothing it away as L2 would. Resize-then-convolve upsampling removes the cause.

The discriminator is discarded and it knew things. After training, the discriminator is thrown away, taking with it a learned model of what real texture looks like in this domain. Several lines of work reuse it as a quality metric or as a component in later fine-tuning, which is worth considering before deleting the checkpoint.