StyleGAN and Style-Based Generation

In 2018 almost every StyleGAN face carried a faint blob somewhere, like a water droplet on the lens. It showed up in the intermediate feature maps from about 64×64 resolution even when the final image hid it, and its cause turned out to be the very operation that made StyleGAN controllable. That arc, from a design that gave unprecedented control to a forensic hunt for the artefacts it introduced, runs through three papers from the same NVIDIA group and is the most detailed public record of how a generator's architecture leaves fingerprints on its images.

The style-based generator

A conventional generator feeds the latent \(z\) into its first layer. StyleGAN does not (Karras, Laine & Aila, 2019, A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR, arXiv:1812.04948). An 8-layer MLP, the mapping network, turns \(z \in \mathbb{R}^{512}\) into an intermediate latent \(w \in \mathbb{R}^{512}\). Synthesis starts from a learned constant \(4 \times 4 \times 512\) tensor, and \(w\) enters every layer through adaptive instance normalisation, borrowed from real-time style transfer (Huang & Belongie, 2017, arXiv:1703.06868):

Each feature map \(x_i\) is normalised on its own, then scaled and shifted by a style \(y = (y_s, y_b)\) that a learned affine transform computes from \(w\). Per-pixel Gaussian noise added after each convolution supplies stochastic detail such as hair placement and freckles.

Why the detour through \(w\)? The sampling distribution of \(z\) is fixed, so the generator must warp it to match the density of real factors of variation, which entangles them. \(W\) is learned and need not follow any fixed distribution, so the mapping network can undo some of that warping. Because styles act per layer, coarse layers govern pose and face shape while fine layers govern colour and texture. Style mixing regularisation, which switches from one \(w\) to another at a random layer for 90% of training images, discourages adjacent layers from assuming their styles are correlated.

The FFHQ ablation shows how much each step contributed. The Progressive GAN baseline scored FID 8.04; tuning brought it to 5.25; adding the mapping network and styles gave 4.85; noise inputs 4.42; mixing regularisation 4.40. The generator has 26.2M parameters against 23.1M for the traditional design. The truncation trick in \(W\), \(w' = \bar{w} + \psi(w - \bar{w})\) with \(\psi < 1\) pulling samples toward the average face, trades variety for average quality at sampling time.

StyleGAN2: the droplet and demodulation

Karras et al. traced the droplet to AdaIN itself (Karras et al., 2020, Analyzing and Improving the Image Quality of StyleGAN, CVPR, arXiv:1912.04958). Normalising each feature map separately destroys information in the relative magnitudes of features. Their hypothesis: the generator sneaks signal strength past the normalisation by creating a strong local spike that dominates the statistics. Removing the normalisation removed the droplets.

The replacement keeps the style's effect but works on expected statistics. Modulation scales the convolution weights by the style, \(w'_{ijk} = s_i \cdot w_{ijk}\), where \(i\) indexes input feature maps, \(j\) output maps and \(k\) spatial position. Assuming unit-variance inputs, output map \(j\) then has standard deviation \(\sigma_j = \sqrt{\sum_{i,k} (w'_{ijk})^2}\), so demodulation divides it out:

This is weaker than instance normalisation, because it relies on an assumption about the signal rather than the actual activations, and that weakness is the point. StyleGAN2 also found that progressive growing made details such as teeth stay aligned to the camera instead of following the head, replaced it with skip and residual architectures, and added path length regularisation for a smoother latent-to-image map. FFHQ FID fell from 4.40 to 2.84.

StyleGAN3: texture sticking and aliasing

Even StyleGAN2 had fine texture glued to pixel coordinates: interpolate the latent so a face moves, and hair and stubble stay put. Karras et al., 2021, Alias-Free Generative Adversarial Networks, NeurIPS, arXiv:2106.12423 blamed positional leaks from image borders, per-pixel noise, positional encodings and, above all, aliasing: upsampling with poor filters and pointwise nonlinearities create frequencies the sample grid cannot represent, and the network learns to exploit the resulting coordinate signal.

The fix treats every feature map as a sampled continuous signal. Fourier features replace the constant input, per-pixel noise goes, upsampling filters are properly designed, and each nonlinearity runs at a temporarily raised resolution before low-pass filtering and downsampling, where a 2× raise proved sufficient. StyleGAN3-R adds rotation equivariance.

When it breaks

FID does not see what the papers were fixing. On aligned FFHQ, StyleGAN2 scored 2.70, StyleGAN3-T 2.79 and StyleGAN3-R 3.07, while translation equivariance rose from 13.58 to above 61 dB. The authors of StyleGAN2 made a similar point, showing generators with identical FID and precision-recall but visibly different quality. Judge these architectures by FID alone and each fix looks neutral or slightly harmful.

Equivariance costs compute. On FFHQ at 1024×1024, training took 1106 GPU-hours for StyleGAN2, 1576 for StyleGAN3-T (+42%) and 2248 for StyleGAN3-R (+103%).

The droplet was a symptom of a clever generator. A network will route information around any normalisation that blocks something it needs, and the resulting artefact can hide below every metric. The general lesson outlived GANs: odd, consistent artefacts in a generator usually point at an architectural constraint the model is working around.

Control is narrow. Style-based editing works best on aligned, single-category data like faces. The same disentanglement does not appear on diverse scenes, where diffusion models with text conditioning displaced the approach.