GANs: The Min-Max Game, Mode Collapse, and Why Training Is Hard

A GAN pits a generator against a discriminator in a minimax game. In theory, the Nash equilibrium is a perfect generative model. In practice, training GANs collapses, oscillates, and diverges in ways that took years of engineering tricks to tame. Understanding why mode collapse happens and what Wasserstein distance fixes makes GAN training legible.

Generative Adversarial Networks (Goodfellow et al., 2014) train a generator G and a discriminator D simultaneously. G maps random noise to data; D tries to distinguish real data from G's output. G tries to fool D. The training objective: min_G max_D E[log D(x)] + E[log(1 - D(G(z)))].

What the Nash equilibrium looks like

At the theoretical optimum, D cannot distinguish real from generated data (D(x) = 0.5 everywhere), and G produces samples from the true data distribution. The discriminator is maximally confused; the generator has perfectly learned the distribution. This is the Nash equilibrium: neither player can improve unilaterally.

Why training is hard: the vanishing gradient problem

When the discriminator is too good early in training, it assigns near-zero probability to generated samples. log(1 - D(G(z))) ≈ log(1) = 0. The generator receives a near-zero gradient — it cannot learn. The practical fix: train the generator to maximise log D(G(z)) rather than minimise log(1 - D(G(z))). These have the same fixed point but the latter has larger gradients early in training.

Mode collapse: the pathological failure mode

Mode collapse occurs when the generator learns to produce only a few high-quality samples rather than covering the full data distribution. Example: when training on a dataset of diverse faces, the generator collapses to producing one or two photorealistic faces repeatedly. Why? G discovers that a narrow set of outputs consistently fools D. Once G commits to these outputs, D adapts to recognise them. G then shifts to a different narrow set. The two networks chase each other through the mode space without converging.

Wasserstein GAN: fixing the loss function

The original GAN loss is equivalent to minimising Jensen-Shannon divergence between the real and generated distributions. JS divergence is 0 when distributions overlap and log 2 when they are disjoint — giving no useful gradient when G is far from the real distribution. Wasserstein GAN (Arjovsky et al., 2017) replaces JS divergence with the Earth Mover distance (Wasserstein-1), which measures the minimum "cost" of transporting one distribution to match the other. It is smooth and provides a useful gradient even when distributions do not overlap. The training objective becomes: min_G max_{||D||_L ≤ 1} E[D(x)] - E[D(G(z))]. The discriminator (now called a critic) must be Lipschitz-constrained (clipping weights or gradient penalty). WGAN training is dramatically more stable.

Progressive growing and StyleGAN

ProGAN (Karras et al., 2018) trains GANs progressively: start at 4×4 resolution, gradually add layers as training stabilises, ending at 1024×1024. Both G and D grow together. Lower resolution is easier to match; early stable training on coarse structure guides later fine-detail learning. StyleGAN (2019) adds style injection at each resolution scale: a mapping network transforms the noise z into a style vector w, which modulates each layer's activations via adaptive instance normalisation. This separates high-level attributes (pose, identity) from fine-grained details (texture, colour), enabling controlled generation and interpolation.

Evaluation: FID and the diversity-quality tradeoff

Fréchet Inception Distance (FID) measures the distance between the distribution of real and generated images in a pretrained Inception network's feature space. Lower FID = better. FID captures both quality (generated samples look realistic) and diversity (generated samples cover the data distribution). A GAN that memorises the training set has low FID. A GAN with mode collapse has high FID despite high per-sample quality. FID is the standard metric but does not fully capture human judgment of generation quality.

Try on Colab: train a DCGAN on CelebA (64×64). After 10 epochs, visualise the discriminator's output distribution for real vs generated images. If the discriminator is too powerful, you should see the generator gradient shrink. Implement gradient penalty (WGAN-GP) and retrain — compare the training loss curves. The Wasserstein loss should be monotonically decreasing rather than oscillating.