Source Summary: Goodfellow et al. 2014 — GANs
Full title: Generative Adversarial Nets
Authors: Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
Affiliation: Université de Montréal
Venue: Advances in Neural Information Processing Systems (NeurIPS) 27 (2014) | arXiv: 1406.2661
PDF: papers/goodfellow-2014-generative-adversarial-nets.pdf
Contributions
- Generative Adversarial Network (GAN) — a framework for training generative models via an adversarial game between a generator G and a discriminator D.
- Minimax game formulation — the training objective is a two-player zero-sum game with a unique Nash equilibrium where G recovers the data distribution and D outputs 1/2 everywhere.
- No MCMC required — GANs generate samples through a single forward pass; no Markov chain Monte Carlo, no variational inference, no explicit likelihood computation.
- Global optimality proof — proved that with unlimited capacity, the unique optimal solution is G(z) ~ p_data and D(x) = 1/2 everywhere.
The Framework
Two networks
- Generator G(z; θ_g): takes noise z ~ p_z(z) as input, outputs a sample in data space. Learns a mapping from noise to realistic data.
- Discriminator D(x; θ_d): takes a data point x (real or fake) as input, outputs a scalar in [0,1] — the probability that x came from real data.
The objective
min_G max_D V(D, G) =
E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 − D(G(z)))]
- D is trained to maximize: assign high scores to real data, low scores to generated data
- G is trained to minimize: fool D into assigning high scores to generated data
This is a minimax game — D and G are adversaries.
Training procedure (Algorithm 1)
Alternate between:
- k steps of D update (k=1 in practice): minimize discriminator loss
- 1 step of G update: maximize G’s ability to fool D
The iterative alternation maintains near-equilibrium between D and G.
Practical G objective
The theoretical objective for G is: minimize E[log(1−D(G(z)))].
In practice: maximize E[log D(G(z))] instead.
Reason: early in training, D easily rejects G’s samples. The gradient of the minimax objective for G saturates (log(1−D(G)) ≈ 0). Maximizing log D(G) provides stronger gradients early in training while having the same fixed point.
Theoretical Analysis
Optimal discriminator
For a fixed G, the optimal discriminator is:
D*(x) = p_data(x) / (p_data(x) + p_g(x))
Global optimum
The minimax game has a unique global optimum: p_g = p_data.
At this point: D*(x) = 1/2 everywhere — the discriminator cannot distinguish real from generated samples.
Training dynamics (convergence)
The paper proves that (under idealized conditions):
- With D at its optimum for a given G, updating G reduces the Jensen-Shannon divergence between p_g and p_data
- The algorithm converges to p_g = p_data
In practice, the proof assumptions don’t hold (finite networks, finite data, alternating optimization), and GAN training is notoriously unstable — but the framework is sound.
Experiments
The original GAN paper trained small MLPs (not CNNs) on:
- MNIST: generated plausible handwritten digits
- TFD (Toronto Face Database): generated face images
- CIFAR-10: early-stage generation of objects
Evaluation: the Parzen window estimate of log-likelihood (a metric that was later found to be unreliable for generative model comparison).
The images in the 2014 paper are crude by modern standards — the key contribution was the framework, not the image quality.
Limitations (Acknowledged)
- Training instability: D and G can fall out of sync; mode collapse (G generates a few samples that fool D but not the full distribution)
- No inference mechanism: unlike VAEs, GANs don’t learn an encoder — you can’t compute latent codes for real data
- Evaluation difficulty: no reliable metric for sample quality
- Mode collapse: G may learn to generate only a subset of the data distribution
Influence
The GAN framework spawned hundreds of variants:
- DCGAN (2015): convolutional GANs, stable training
- cGAN (2014): conditional generation on class labels
- CycleGAN (2017): unpaired image-to-image translation
- StyleGAN (2018/2019): high-fidelity face generation (NVIDIA)
- BigGAN (2018): large-scale class-conditional generation
GANs dominated generative modeling until Diffusion models (2020–2022) surpassed them in image quality and training stability.
Key References
- Hinton et al. 2006 — Restricted Boltzmann Machines (prior generative model)
- Kingma & Welling 2013 — Variational Autoencoders (concurrent generative approach)
- Rumelhart et al. 1986 — backpropagation (training mechanism)
- Ho et al. 2020 — Denoising Diffusion Models (successor paradigm)