Generative Adversarial Networks

The variational autoencoder learns to generate by maximizing a likelihood bound; it pays for tractability with blurry samples, because a pixel-wise reconstruction loss favors averaged, hedged predictions. A generative adversarial network (GAN) discards the likelihood entirely and replaces it with a learned critic. Instead of scoring how probable an image is under the model, it trains the generator against a second network that tries to tell generated images from real ones, and the generator's goal is to make that network fail. There is no explicit density, no reconstruction term, only a game.¹

The generator never sees the data. It only ever receives a gradient routed back through the discriminator, the single channel that carries any information about what real samples look like.

The minimax game

Write $p_{data}$ for the (unknown) data distribution and $p_g$ for the distribution of $G_{\theta }(z)$ induced by pushing the noise prior through the generator. The discriminator should output near $1$ on real samples and near $0$ on fakes; the generator wants the opposite for its own outputs. Both goals are captured by a single value function, the binary cross-entropy of the discriminator's classification task:

The discriminator maximizes $V$ (push $D(x)\to 1$ on real, $D(G(z))\to 0$ on fake); the generator minimizes it (push $D(G(z))\to 1$). The training objective is the saddle point:

The optimal discriminator

Fix the generator, hence fix $p_g$, and solve the inner maximization over $D$. This is a pointwise problem: $V$ is an expectation over $x$ of a function that depends on $x$ only through the scalar $D(x)$, so we can optimize the integrand at each $x$ independently.

The optimal discriminator is not a black box² but a recognizable statistical object: a calibrated posterior probability. Where the two densities are equal it returns exactly $\frac{1}{2}$ — it has no information and guesses.

What the generator actually minimizes

Substitute $D^{\ast }$ back into $V$ to find the objective the generator faces once the discriminator is optimal. This is the central result of the GAN theory: the adversarial game reduces to a divergence between distributions.

At equilibrium the discriminator is reduced to a coin flip: it cannot do better than chance because the two distributions coincide. Compare the divergence GANs minimize against the one VAEs and likelihood models minimize: the asymmetry of the choice is the source of GANs' signature behavior.

The JSD is symmetric and bounded by $\log 2$, and that boundedness is what lets GANs produce sharp samples while quietly abandoning parts of the data, a trade we return to under mode collapse.³

A worked objective computation

For example, take a single point $x_0$ where the data density is $p_{data}(x_0)=0.8$ and the generator's is $p_g(x_0)=0.2$. The optimal discriminator there is the likelihood ratio

It correctly leans toward real, since data is four times as dense there. The value-function integrand at $x_0$ (per unit of the two densities) is

Now step back to a whole toy distribution and compute the generator's true objective $C(G)=-\log 4+2\mathrm{JSD}$. Let both densities live on two points $\{a,b\}$ with

so the mixture is $m=(0.65,0.35)$. The two KL divergences to the mixture are

So $\mathrm{JSD}=\frac{1}{2}(0.0473+0.0544)=0.0509$ nats, and the generator's objective sits at

The floor is $-\log 4=-1.386$, reached only when $p_g=p_{data}$. The generator is $0.102$ nats above the floor — a small, bounded gap, which is the whole point: even a fairly wrong generator ($0.8$ versus $0.5$ on one atom) incurs only a modest JSD penalty. That boundedness is precisely why the generator can afford to sit on one mode, the seed of mode collapse.

The training dynamics

In practice we cannot reach $D^{\ast }$ before each generator step; that would require an inner optimization to convergence per outer step. GANs instead alternate a few discriminator gradient ascents with a generator gradient descent, keeping the two roughly in step. With $k$ discriminator updates per generator update (often $k=1$):

As training proceeds, $p_g$ is dragged toward $p_{data}$ and the discriminator's decision curve $D(x)$ flattens from a confident step toward the constant $\frac{1}{2}$ — the visual signature of convergence in Goodfellow's original figure.

Why the original generator loss saturates

The minimax form has $G$ minimize $\log (1-D(G(z)))$. Early in training $G$ is weak, the discriminator confidently rejects every fake, and $D(G(z))\approx 0$. Examine the gradient of that loss with respect to the discriminator output $d=D(G(z))$:

At $d\to 0$ this is $\approx -1$, small. The function $\log (1-d)$ is flat near $d=0$ and only steepens as $d\to 1$. So precisely when the generator is losing badly (and most needs a strong corrective signal) the gradient is weakest; the loss saturates. The fix is to give the generator a different objective with the same fixed point but a healthy gradient where it matters.

The non-saturating loss is what every practical GAN actually trains with. It no longer corresponds to a clean JSD minimization (it minimizes a different divergence combination), but it makes the early gradient usable, which matters far more than theoretical purity.⁴

Failure modes

GAN training is a search for a saddle point of a non-convex two-player game, not a descent to a minimum. There is no monotone objective whose decrease certifies progress, and three pathologies follow directly.

Mode collapse

The JSD is mode-seeking: a generator that places all its mass on a single sharp mode of $p_{data}$ already achieves a low (though not optimal) divergence, because JSD does not heavily punish missing mass the way forward KL does. If the discriminator cannot momentarily distinguish a single perfectly-rendered mode from the data, the generator has every incentive to collapse onto it.⁵

Fixes that change the objective

The deepest fix is to replace the JSD with a divergence whose gradient is informative even when $p_{data}$ and $p_g$ have disjoint support, the regime where JSD is constant at $\log 2$ and supplies no gradient at all.

Latent structure

Because $G$ is a smooth, deterministic map from a low-dimensional noise space, the geometry of $\mathcal{Z}$ becomes a geometry of generated samples. Interpolating along a straight line between two noise vectors $z_0$ and $z_1$ produces a continuous morph between $G(z_0)$ and $G(z_1)$, evidence that the generator has learned a meaningful manifold rather than memorizing examples.⁶

The GAN decade

Goodfellow's 2014 paper launched a fast-moving line of work, and a few landmarks explain why GANs dominated image synthesis until diffusion arrived.⁷

Architecture came first. The original GAN used fully connected nets on small images; the DCGAN established the convolutional recipe — strided convolutions, batch normalization, no pooling — that made GAN training reproducible and is the template Chollet's practical chapter follows.⁸ Scaling then came from progressive growing and StyleGAN, whose style-based generator gave the disentangled, editable latent space behind the photorealistic faces that made GANs famous.⁹

Theory chased the instability. The Wasserstein GAN reframed the objective around earth-mover distance so gradients survive disjoint support, and WGAN-GP replaced weight clipping with a gradient penalty to enforce the Lipschitz constraint cleanly.¹⁰ Spectral normalization bounded the discriminator's Lipschitz constant a different way, and the two time-scale update rule with the Fréchet Inception Distance gave the field both a stable training scheme and the sample-quality metric it had been missing.¹¹

Conditioning made GANs practical. Conditional GANs feed a label to both networks; pix2pix and CycleGAN turned the framework into image-to-image translation, learning to map sketches to photos or horses to zebras — CycleGAN doing so without paired data, using a cycle-consistency loss in place of the pairing.¹² These are the applications, not the theory, but they are why GANs left the lab.

The learned-critic idea outlived GANs' reign as the top image model: it survives as the perceptual and adversarial losses inside super-resolution networks, as the discriminator that sharpens VAE and diffusion decoders, and as the training signal in the codec models that compress the latents diffusion runs on.

Where GANs sit among generative models

GANs are one of four major families of deep generative model, and the four trade off along three axes: whether they offer a tractable likelihood, how good their samples look, and how stable they are to train. Each sibling lesson develops one column.

The dividing line is the likelihood. Autoregressive models and flows compute $p_{\theta }(x)$ exactly and train by maximum likelihood; VAEs settle for a bound; GANs abandon it entirely, gaining sharpness at the cost of a likelihood they can neither report nor optimize. The trade-off — no density, no easy evaluation, frequent training failures — yields the sharpest samples of any pre-diffusion family, and a learned latent space smooth enough to interpolate through.