Conditional GANs and Image-to-Image Translation

Give a GAN a label map of a street and ask for a photograph. Any realistic photograph will fool an unconditional discriminator, including one of a different street. Conditioning exists to close that loophole: the discriminator must judge not just "is this real?" but "is this real and does it match the input?". How the condition is shown to the discriminator turns out to matter as much as whether it is shown at all.

The conditional objective

The original conditional GAN feeds the condition \(y\) to both networks, typically by concatenating an embedding (Mirza & Osindero, 2014, Conditional Generative Adversarial Nets, arXiv:1411.1784):

The minimax game itself is unchanged from the unconditional case covered in its own concept. What changes is the optimum: the generator now matches \(p(x \mid y)\) for each \(y\), and a sample that is realistic but wrong for its \(y\) is penalised.

Paired translation: pix2pix

When training pairs \((x, y)\) exist, such as edges and photos or maps and aerial views, pix2pix combines the conditional adversarial loss with a reconstruction term (Isola, Zhu, Zhou & Efros, 2017, Image-to-Image Translation with Conditional Adversarial Networks, CVPR, arXiv:1611.07004):

The L1 term pins low-frequency structure; the adversarial term, through a patch discriminator described in the concept on adversarial losses as a component, restores sharp texture. The generator is a U-Net whose skip connections let fine input detail such as edge positions bypass the bottleneck.

One detail contradicts the textbook picture. Past conditional GANs fed Gaussian noise \(z\) to the generator for stochasticity; pix2pix's generator learned to ignore it. The final model has noise only as dropout at training and test time, and the authors report only minor stochasticity. Conditional GANs with a strong input readily collapse to a nearly deterministic map, which is fine for rendering a label map and wrong for anything with genuinely many plausible outputs.

Unpaired translation: CycleGAN

Horse photographs and zebra photographs exist; photographs of the same horse as a zebra do not. CycleGAN learns two generators, \(G: X \to Y\) and \(F: Y \to X\), each with an adversarial loss against its target domain, and ties them with cycle consistency (Zhu, Park, Isola & Efros, 2017, Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV, arXiv:1703.10593):

weighted by \(\lambda = 10\). The adversarial terms make outputs look like the target domain; the cycle term forbids discarding the input's content, since whatever \(G\) throws away \(F\) cannot restore. Training therefore runs four networks, two generators and two discriminators, roughly doubling the cost of a paired setup.

Showing the condition to the discriminator

Concatenating \(y\) to the discriminator's input or hidden features is arbitrary: nothing tells the network how the condition relates to the likelihood ratio it is estimating. Miyato and Koyama started from that ratio. If the conditional distributions are log-linear in features \(\phi(x)\), the optimal discriminator's logit takes the form

where \(y\) is a one-hot label, \(V\) an embedding matrix, and \(\psi\) a scalar head for the unconditional term (Miyato & Koyama, 2018, cGANs with Projection Discriminator, ICLR, arXiv:1802.05637). The label enters through an inner product with the features instead of being concatenated. On 1000-class ImageNet, intra-class FID was 260.0 for AC-GAN, 141.2 for hidden-layer concatenation and 103.1 for projection, falling to 92.4 with longer training, and AC-GAN's reported number is from before its training collapsed. Projection became the default conditioning in large class-conditional GANs, and later diffusion distillation work reused it to condition discriminators.

When it breaks

Cycle consistency can be satisfied by cheating. Chu, Zhmoginov and Sandler showed CycleGAN hides information about the source image in near-imperceptible high-frequency signals in its output, which \(F\) decodes to reconstruct the input (Chu et al., 2017, CycleGAN, a Master of Steganography, arXiv:1712.02950). The loss is minimised; the translation is not semantically faithful, and the hidden signal behaves like an adversarial perturbation.

Geometry does not translate. CycleGAN's authors report success on colour and texture changes and little success on shape changes. Dog-to-cat degenerated into minimal edits of the input. A pixel-wise cycle loss rewards staying close to the input, which is the opposite of what a geometric change needs.

Dataset gaps become hallucinations. Trained on ImageNet horses and zebras without riders, CycleGAN painted zebra stripes over a person riding a horse. Unpaired methods learn whatever statistics separate the two collections, including ones you did not intend.

One-to-one by construction. Both pix2pix's ignored noise and CycleGAN's deterministic inverse push toward a single output per input. Tasks with many valid answers, such as colourising a grey car, need explicit multimodal designs.