--- title: Generative Adversarial Networks module: Generative Models moduleNumber: 7 lessonNumber: 4 order: 704 summary: > A generative adversarial network trains two networks against each other: a generator that turns noise into samples, and a discriminator that tries to tell real data from forgeries. The game has a clean theory: the optimal discriminator is a likelihood ratio, and at equilibrium the generator minimizes the Jensen–Shannon divergence to the data, with a global optimum exactly when its distribution matches the data. We derive that result, fix the saturating loss that breaks training, and catalogue the failure modes (mode collapse, instability, vanishing gradients) and the architectural fixes. topics: [Generative Models] sources: - book: Goodfellow ref: "§20.10.4 — Generative Adversarial Networks" - book: Goodfellow ref: "§3.13 — Information Theory (KL & cross-entropy)" - book: Chollet ref: "§8.5 — Introduction to Generative Adversarial Networks" --- The [variational autoencoder](/deep-learning/generative-models/variational-autoencoders) learns to generate by maximizing a likelihood bound; it pays for tractability with blurry samples, because a pixel-wise reconstruction loss favors averaged, hedged predictions. A **generative adversarial network** (GAN) discards the likelihood entirely and replaces it with a learned critic. Instead of scoring how probable an image is under the model, it trains the generator against a second network that tries to tell generated images from real ones, and the generator's goal is to make that network fail. There is no explicit density, no reconstruction term, only a game.[^gf-gan] > **Definition (Generative adversarial network).** A pair of networks trained in > opposition: a **generator** $G_{\theta}: \mathcal{Z} \to \mathcal{X}$ mapping a > noise vector $z \sim p(z)$ to a sample $G_{\theta}(z)$, and a **discriminator** > $D_{\phi}: \mathcal{X} \to (0,1)$ outputting the probability that its input is > real data rather than a generated forgery. $G$ is trained to fool $D$; $D$ is > trained not to be fooled. The generator never sees the data. It only ever receives a gradient routed back through the discriminator, the single channel that carries any information about what real samples look like. $$ % caption: The GAN architecture: generator $G$ turns noise into fakes; discriminator $D$ scores real and fake samples, and its gradient is $G$'s only learning signal. \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=22mm, minimum height=12mm, align=center}, src/.style={draw, minimum width=20mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} % noise -> generator -> fake \node[src] (z) at (0,-1.6) {noise $z$}; \node[box, draw=acc, text=acc, thick] (G) at (3.1,-1.6) {generator $G$}; \node[src] (fake) at (6.4,-1.6) {\texttt{fake} $G(z)$}; % real data source \node[src] (real) at (6.4,1.0) {real $x$}; % discriminator \node[box, draw=red, text=red, thick] (D) at (9.6,-0.3) {discriminator $D$}; \node[src] (out) at (12.9,-0.3) {real / \texttt{fake}}; \draw[->, acc, thick] (z) -- (G); \draw[->, acc, thick] (G) -- (fake); \draw[->, green, thick] (real) -- (D); \draw[->, acc, thick] (fake) -- (D); \draw[->, red, thick] (D) -- (out); % adversarial feedback loop: gradient back to G \draw[->, black, thick, dashed] (D.south) .. controls (9.6,-3.0) and (3.1,-3.4) .. (G.south) node[pos=0.5, below, font=\footnotesize, text=black] {\texttt{gradient} back to $G$}; \end{tikzpicture} $$ ## The minimax game Write $p_{data}$ for the (unknown) data distribution and $p_g$ for the distribution of $G_{\theta}(z)$ induced by pushing the noise prior through the generator. The discriminator should output near $1$ on real samples and near $0$ on fakes; the generator wants the opposite for its own outputs. Both goals are captured by a single **value function**, the binary cross-entropy of the discriminator's classification task: $$ V(D, G) \;=\; \mathbb{E}_{x \sim p_{data}}\!\brackets{\log D(x)} \;+\; \mathbb{E}_{z \sim p(z)}\!\brackets{\log\parens{1 - D(G(z))}}. $$ The discriminator maximizes $V$ (push $D(x) \to 1$ on real, $D(G(z)) \to 0$ on fake); the generator minimizes it (push $D(G(z)) \to 1$). The training objective is the saddle point: $$ \min_{G} \, \max_{D} \; V(D, G). $$ > **Definition (Two-player minimax game).** A zero-sum game in which one player > maximizes a value function and the other minimizes the _same_ function. The > GAN's value $V(D,G)$ is the discriminator's expected log-likelihood on a balanced > real/fake classification problem; $D$ climbs it and $G$ descends it. $$ % caption: The minimax game: two players push the value function $V(D,G)$ in opposite directions, with equilibrium at the saddle point. \begin{tikzpicture}[>=stealth, font=\footnotesize, player/.style={draw, minimum width=26mm, minimum height=13mm, align=center}] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \definecolor{red}{HTML}{C0392B} \node[player, draw=green, text=green, thick] (D) at (-4.3,0) {discriminator\\maximizes $V$}; \node[draw, minimum width=22mm, minimum height=15mm] (V) at (0,0) {\texttt{V(D, G)}}; \node[player, draw=acc, text=acc, thick] (G) at (4.3,0) {generator\\minimizes $V$}; \draw[->, green, very thick] (D) -- (V) node[pos=0.5, above, font=\scriptsize, text=green] {push up}; \draw[->, acc, very thick] (G) -- (V) node[pos=0.5, above, font=\scriptsize, text=acc] {push down}; \node[font=\footnotesize, text=black, align=center] at (0,-1.6) {equilibrium = saddle \texttt{point}}; \end{tikzpicture} $$ ## The optimal discriminator Fix the generator, hence fix $p_g$, and solve the inner maximization over $D$. This is a pointwise problem: $V$ is an expectation over $x$ of a function that depends on $x$ only through the scalar $D(x)$, so we can optimize the integrand at each $x$ independently. > **Theorem (Optimal discriminator).** For a fixed generator $G$, the discriminator > that maximizes $V(D,G)$ is the likelihood ratio > $$ > D^{\ast}(x) \;=\; \frac{p_{data}(x)}{p_{data}(x) + p_g(x)}. > $$ > **Proof.** Change the second expectation from noise space to sample space: since > $G$ pushes $p(z)$ forward to $p_g$, $\mathbb{E}_{z}[\log(1 - D(G(z)))] = > \mathbb{E}_{x \sim p_g}[\log(1 - D(x))]$. Then > $$ > V(D,G) = \int_{x} \parens{ p_{data}(x)\,\log D(x) + p_g(x)\,\log\parens{1 - D(x)} }\, dx. > $$ > The integrand has the form $a \log d + b \log(1 - d)$ in $d = D(x)$, with > $a = p_{data}(x) \ge 0$ and $b = p_g(x) \ge 0$. Differentiate and set to zero: > $$ > \frac{\partial}{\partial d}\brackets{a \log d + b \log(1-d)} > = \frac{a}{d} - \frac{b}{1-d} = 0 > \;\Longrightarrow\; > a(1-d) = b\,d > \;\Longrightarrow\; > d = \frac{a}{a+b}. > $$ > The second derivative $-a/d^2 - b/(1-d)^2 < 0$, so this is the maximum. Substituting > $a, b$ gives $D^{\ast}(x) = p_{data}(x)/(p_{data}(x) + p_g(x))$ at every $x$. $\qed$ The optimal discriminator is not a black box[^gf-optd] but a recognizable statistical object: a calibrated posterior probability. Where the two densities are equal it returns exactly $\tfrac12$ — it has no information and guesses. | Region | $p_{data}$ vs $p_g$ | $D^{\ast}(x)$ | Meaning | | --- | --- | --- | --- | | data-dominated | $p_{data} \gg p_g$ | $\to 1$ | confidently real | | matched | $p_{data} = p_g$ | $\tfrac12$ | indistinguishable | | fake-dominated | $p_g \gg p_{data}$ | $\to 0$ | confidently fake | ## What the generator actually minimizes Substitute $D^{\ast}$ back into $V$ to find the objective the generator faces once the discriminator is optimal. This is the central result of the GAN theory: the adversarial game reduces to a divergence between distributions. > **Theorem (Global optimum and the Jensen–Shannon divergence).** When $D = D^{\ast}$, > the generator's objective is > $$ > C(G) = \max_{D} V(D, G) = -\log 4 \;+\; 2\,\mathrm{JSD}\parens{p_{data} \,\|\, p_g}, > $$ > where $\mathrm{JSD}$ is the Jensen–Shannon divergence. Since $\mathrm{JSD} \ge 0$ > with equality iff its arguments are equal, $C(G)$ attains its global minimum > $-\log 4$ exactly when $p_g = p_{data}$. > **Proof.** Write $m(x) = \tfrac12\parens{p_{data}(x) + p_g(x)}$, so > $D^{\ast}(x) = p_{data}/(2m)$ and $1 - D^{\ast}(x) = p_g/(2m)$. Then > $$ > C(G) = \mathbb{E}_{p_{data}}\!\brackets{\log \tfrac{p_{data}}{2m}} > + \mathbb{E}_{p_g}\!\brackets{\log \tfrac{p_g}{2m}}. > $$ > Split each logarithm, $\log \tfrac{p}{2m} = \log \tfrac{p}{m} - \log 2$. The two > $-\log 2$ terms collect to $-\log 4$, leaving two KL divergences against the > mixture $m$: > $$ > C(G) = -\log 4 + \mathrm{KL}\parens{p_{data} \,\|\, m} + \mathrm{KL}\parens{p_g \,\|\, m}. > $$ > By definition $\mathrm{JSD}(p \,\|\, q) = \tfrac12 \mathrm{KL}(p \,\|\, m) + > \tfrac12 \mathrm{KL}(q \,\|\, m)$ with $m = \tfrac12(p+q)$, so the two KL terms > equal $2\,\mathrm{JSD}(p_{data} \,\|\, p_g)$. Hence $C(G) = -\log 4 + > 2\,\mathrm{JSD}(p_{data} \,\|\, p_g)$. The JSD is nonnegative and vanishes iff > $p_{data} = p_g$, giving the unique global minimum at $p_g = p_{data}$, where > $D^{\ast} \equiv \tfrac12$. $\qed$ At equilibrium the discriminator is reduced to a coin flip: it cannot do better than chance because the two distributions coincide. Compare the divergence GANs minimize against the one VAEs and likelihood models minimize: the asymmetry of the choice is the source of GANs' signature behavior. | Objective | Divergence minimized | Behavior when $p_g$ misses a mode | | --- | --- | --- | | maximum likelihood / VAE | $\mathrm{KL}(p_{data} \,\|\, p_g)$ | heavily penalized — forces mass everywhere ("mode covering") | | GAN (ideal) | $\mathrm{JSD}(p_{data} \,\|\, p_g)$ | symmetric, bounded — tolerates dropping modes ("mode seeking") | | reverse KL | $\mathrm{KL}(p_g \,\|\, p_{data})$ | penalizes putting mass where data is absent | The JSD is symmetric and bounded by $\log 2$, and that boundedness is what lets GANs produce sharp samples while quietly abandoning parts of the data, a trade we return to under mode collapse.[^gf-jsd] ### A worked objective computation For example, take a single point $x_0$ where the data density is $p_{data}(x_0) = 0.8$ and the generator's is $p_g(x_0) = 0.2$. The optimal discriminator there is the likelihood ratio $$ D^{\ast}(x_0) = \frac{p_{data}(x_0)}{p_{data}(x_0) + p_g(x_0)} = \frac{0.8}{0.8 + 0.2} = 0.8. $$ It correctly leans toward "real," since data is four times as dense there. The value-function integrand at $x_0$ (per unit of the two densities) is $$ p_{data}\log D^{\ast} + p_g\log(1 - D^{\ast}) = 0.8\log 0.8 + 0.2\log 0.2 = -0.178 - 0.322 = -0.500. $$ Now step back to a whole toy distribution and compute the generator's true objective $C(G) = -\log 4 + 2\,\mathrm{JSD}$. Let both densities live on two points $\{a, b\}$ with $$ p_{data} = (0.5,\; 0.5), \qquad p_g = (0.8,\; 0.2), $$ so the mixture is $m = (0.65,\; 0.35)$. The two KL divergences to the mixture are $$ \begin{aligned} \mathrm{KL}(p_{data}\,\|\,m) &= 0.5\log\tfrac{0.5}{0.65} + 0.5\log\tfrac{0.5}{0.35} = -0.131 + 0.178 = 0.0473, \\ \mathrm{KL}(p_g\,\|\,m) &= 0.8\log\tfrac{0.8}{0.65} + 0.2\log\tfrac{0.2}{0.35} = 0.166 - 0.112 = 0.0544. \end{aligned} $$ So $\mathrm{JSD} = \tfrac12(0.0473 + 0.0544) = 0.0509$ nats, and the generator's objective sits at $$ C(G) = -\log 4 + 2(0.0509) = -1.386 + 0.102 = -1.284. $$ The floor is $-\log 4 = -1.386$, reached only when $p_g = p_{data}$. The generator is $0.102$ nats above the floor — a small, bounded gap, which is the whole point: even a fairly wrong generator ($0.8$ versus $0.5$ on one atom) incurs only a modest JSD penalty. That boundedness is precisely why the generator can afford to sit on one mode, the seed of mode collapse. $$ % caption: Fitting a single Gaussian $p_g$ to a two-mode target. Forward KL (mode-covering) spreads $p_g$ across both bumps; JSD (mode-seeking) locks onto one, sharp but incomplete. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % --- left: mode covering (KL) --- \begin{scope} \draw[->, thick] (-0.2,0) -- (5.0,0) node[right, font=\scriptsize] {$x$}; \draw[->, thick] (0,-0.2) -- (0,2.7); % two-mode target (green) \draw[green, very thick] plot[domain=0.1:4.8, samples=90] (\x, {1.5*exp(-0.5*(\x-1.3)*(\x-1.3)/0.2) + 1.5*exp(-0.5*(\x-3.4)*(\x-3.4)/0.2)}); % single broad gaussian spanning both (blue) \draw[acc, very thick] plot[domain=0.1:4.8, samples=90] (\x, {1.35*exp(-0.5*(\x-2.35)*(\x-2.35)/1.3)}); \node[font=\footnotesize, anchor=north] at (2.4,-0.35) {\texttt{forward} KL: covers}; \end{scope} % --- right: mode seeking (JSD) --- \begin{scope}[xshift=6.6cm] \draw[->, thick] (-0.2,0) -- (5.0,0) node[right, font=\scriptsize] {$x$}; \draw[->, thick] (0,-0.2) -- (0,2.7); \draw[green, very thick] plot[domain=0.1:4.8, samples=90] (\x, {1.5*exp(-0.5*(\x-1.3)*(\x-1.3)/0.2) + 1.5*exp(-0.5*(\x-3.4)*(\x-3.4)/0.2)}); % sharp gaussian on one mode (blue) \draw[acc, very thick] plot[domain=0.1:4.8, samples=90] (\x, {1.5*exp(-0.5*(\x-3.4)*(\x-3.4)/0.2)}); \node[font=\scriptsize, anchor=north] at (2.4,-0.35) {JSD: seeks one}; \end{scope} \end{tikzpicture} $$ ## The training dynamics In practice we cannot reach $D^{\ast}$ before each generator step; that would require an inner optimization to convergence per outer step. GANs instead **alternate** a few discriminator gradient ascents with a generator gradient descent, keeping the two roughly in step. With $k$ discriminator updates per generator update (often $k=1$): ```algorithm caption: $\textsc{TrainGAN}(p_{data}, p(z), k, \eta)$ — alternating adversarial updates initialize generator params $\theta$ and discriminator params $\phi$ repeat for $k$ steps do // train the discriminator sample minibatch $x_1, \dots, x_m \sim p_{data}$ sample minibatch $z_1, \dots, z_m \sim p(z)$ $g_\phi \gets \nabla_\phi \tfrac1m \sum_i \brackets{\log D(x_i) + \log(1 - D(G(z_i)))}$ $\phi \gets \phi + \eta \cdot g_\phi$ // ascend: $D$ maximizes $V$ sample minibatch $z_1, \dots, z_m \sim p(z)$ $g_\theta \gets \nabla_\theta \tfrac1m \sum_i \log D(G(z_i))$ // non-saturating loss $\theta \gets \theta + \eta \cdot g_\theta$ // ascend the surrogate until converged return $G_\theta$ ``` As training proceeds, $p_g$ is dragged toward $p_{data}$ and the discriminator's decision curve $D(x)$ flattens from a confident step toward the constant $\tfrac12$ — the visual signature of convergence in Goodfellow's original figure. $$ % caption: Distributions converging over training: as $p_g$ (blue) approaches $p_{data}$ (green), the optimal $D$ (black, dashed) flattens toward $\tfrac12$. \begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} % ===== EARLY (left) ===== \begin{scope} \draw[->, thick] (-0.3,0) -- (5.4,0) node[right, font=\scriptsize] {$x$}; \draw[->, thick] (0,-0.2) -- (0,2.7); % data density (green), centered at 3.4 \draw[green, very thick] plot[domain=0.2:5.2, samples=70] (\x, {1.9*exp(-0.5*(\x-3.4)*(\x-3.4)/0.45)}); % generator density (blue), displaced to 1.6 \draw[acc, very thick] plot[domain=0.2:5.2, samples=70] (\x, {1.9*exp(-0.5*(\x-1.6)*(\x-1.6)/0.45)}); % half line \draw[black, dashed] (-0.3,1.1) -- (5.2,1.1); \node[black, font=\scriptsize, anchor=east] at (-0.05,1.1) {$\tfrac12$}; % discriminator curve: sharp transition near x=2.5 (sigmoid-like), scaled to ~[0.1,2.1] \draw[black, thick, dashed] plot[domain=0.2:5.2, samples=70] (\x, {0.2 + 1.8/(1+exp(-4*(\x-2.5)))}); \node[font=\scriptsize, align=center] at (2.5,-0.7) {early: $D$ separates}; \node[green, font=\scriptsize] at (3.7,2.25) {$p_{data}$}; \node[acc, font=\scriptsize] at (1.3,2.25) {$p_g$}; \end{scope} % ===== LATE (right) ===== \begin{scope}[xshift=7.2cm] \draw[->, thick] (-0.3,0) -- (5.4,0) node[right, font=\scriptsize] {$x$}; \draw[->, thick] (0,-0.2) -- (0,2.7); % both densities near 2.9, nearly overlapping \draw[green, very thick] plot[domain=0.2:5.2, samples=70] (\x, {1.9*exp(-0.5*(\x-2.95)*(\x-2.95)/0.5)}); \draw[acc, very thick, dash pattern=on 3pt off 2pt] plot[domain=0.2:5.2, samples=70] (\x, {1.85*exp(-0.5*(\x-2.85)*(\x-2.85)/0.52)}); \draw[black, dashed] (-0.3,1.1) -- (5.2,1.1); \node[black, font=\scriptsize, anchor=east] at (-0.05,1.1) {$\tfrac12$}; % discriminator flat near 1/2 \draw[black, thick, dashed] plot[domain=0.2:5.2, samples=70] (\x, {1.1 + 0.12*sin(\x*60)}); \node[font=\scriptsize, align=center] at (2.5,-0.7) {converged: $D$ near $\tfrac12$}; \end{scope} \end{tikzpicture} $$ ### Why the original generator loss saturates The minimax form has $G$ _minimize_ $\log(1 - D(G(z)))$. Early in training $G$ is weak, the discriminator confidently rejects every fake, and $D(G(z)) \approx 0$. Examine the gradient of that loss with respect to the discriminator output $d = D(G(z))$: $$ \frac{\partial}{\partial d}\,\log(1 - d) = -\frac{1}{1 - d}. $$ At $d \to 0$ this is $\approx -1$, small. The function $\log(1-d)$ is _flat_ near $d=0$ and only steepens as $d \to 1$. So precisely when the generator is losing badly (and most needs a strong corrective signal) the gradient is weakest; the loss **saturates**. The fix is to give the generator a different objective with the same fixed point but a healthy gradient where it matters. > **Definition (Non-saturating generator loss).** Rather than minimizing > $\log(1 - D(G(z)))$, the generator _maximizes_ $\log D(G(z))$, i.e. minimizes > $-\log D(G(z))$. Both objectives share the same optimum ($D(G(z)) \to 1$), but the > non-saturating form has gradient $\partial_d \log d = 1/d$, which _blows up_ as > $d \to 0$: strongest exactly when $G$ is losing. $$ % caption: The two generator losses versus $d = D(G(z))$: the minimax loss (black) is flat near $d=0$, while the non-saturating loss (blue) stays steep there. \begin{tikzpicture}[>=stealth, font=\footnotesize, scale=1.0] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (-0.2,0) -- (5.6,0) node[right, font=\scriptsize] {$d = D(G(z))$}; \draw[->, thick] (0,-0.2) -- (0,3.2) node[above, font=\scriptsize] {loss}; % x maps d in [0,1] to [0,5]; so d = x/5 % non-saturating: -log d (clamped), steep near d=0 \draw[acc, very thick] plot[domain=0.45:5.0, samples=80] (\x, {-1.3*ln(\x/5)}); \node[acc, font=\scriptsize, anchor=north west] at (1.55,2.95) {non-saturating}; % minimax: log(1-d), flat near 0, dives near d=1 -> show its magnitude rising near d->1 \draw[black, very thick] plot[domain=0.05:4.55, samples=80] (\x, {-1.3*ln(1-\x/5)}); \node[black, font=\scriptsize, anchor=south east] at (4.45,1.6) {saturating}; \draw[black, dashed] (0,0) -- (0,3.0); \node[font=\scriptsize, text=black, anchor=north] at (0.55,-0.25) {$G$ losing}; \node[font=\scriptsize, text=black, anchor=north] at (4.8,-0.25) {$G$ winning}; \end{tikzpicture} $$ The non-saturating loss is what every practical GAN actually trains with. It no longer corresponds to a clean JSD minimization (it minimizes a different divergence combination), but it makes the early gradient usable, which matters far more than theoretical purity.[^gf-nonsat] ## Failure modes GAN training is a search for a saddle point of a non-convex two-player game, not a descent to a minimum. There is no monotone objective whose decrease certifies progress, and three pathologies follow directly. | Failure mode | Mechanism | Symptom | Cure | | --- | --- | --- | --- | | Mode collapse | $G$ maps many $z$ to few outputs that reliably fool $D$ | low sample diversity; whole modes missing | feature matching, minibatch discrimination, unrolled GAN | | Vanishing gradients | $D$ wins early; $D(G(z)) \to 0$ on a flat loss region | $G$ stops improving | non-saturating loss; Wasserstein (WGAN) | | Instability / oscillation | the saddle-point dynamics cycle instead of converging | losses oscillate; samples flicker | spectral normalization; two time-scale update (TTUR); gradient penalty | ### Mode collapse The JSD is _mode-seeking_: a generator that places all its mass on a single sharp mode of $p_{data}$ already achieves a low (though not optimal) divergence, because JSD does not heavily punish missing mass the way forward KL does. If the discriminator cannot momentarily distinguish a single perfectly-rendered mode from the data, the generator has every incentive to collapse onto it.[^gf-collapse] $$ % caption: Mode collapse: the target $p_{data}$ has four modes (green), but the generator's samples (blue) pile onto a single one, ignoring the rest. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{green}{HTML}{1F9D4D} \draw[->, thick] (-0.3,0) -- (6.6,0) node[right, font=\scriptsize] {$x_1$}; \draw[->, thick] (0,-0.3) -- (0,5.0) node[above, font=\scriptsize] {$x_2$}; % four target clusters (green rings of points) \foreach \cx/\cy in {1.4/3.6, 5.0/3.8, 1.6/1.2, 5.2/1.3} { \foreach \dx/\dy in {0.0/0.0, 0.35/0.2, -0.3/0.25, 0.2/-0.35, -0.25/-0.25, 0.4/-0.1} { \fill[green!75] (\cx+\dx,\cy+\dy) circle (1.6pt); } \draw[green, thick, dashed] (\cx,\cy) circle (0.62); } % generator samples: dense pile on one mode (bottom-left cluster) \foreach \dx/\dy in {0.0/0.0, 0.18/0.12, -0.15/0.16, 0.12/-0.2, -0.18/-0.12, 0.24/0.04, -0.22/0.0, 0.05/0.22, 0.1/-0.1, -0.08/-0.2, 0.2/-0.05, -0.1/0.1} { \fill[acc] (1.6+\dx,1.2+\dy) circle (2.0pt); } \node[acc, font=\scriptsize, anchor=north] at (1.6,0.35) {all $G$ samples here}; \node[green, font=\footnotesize, anchor=south] at (5.0,4.55) {target \texttt{modes}\ (ignored)}; \end{tikzpicture} $$ ### Fixes that change the objective The deepest fix is to replace the JSD with a divergence whose gradient is informative even when $p_{data}$ and $p_g$ have disjoint support, the regime where JSD is constant at $\log 2$ and supplies no gradient at all. > **Definition (Wasserstein GAN).** Replace the value function with the Wasserstein-1 > (earth-mover) distance $W(p_{data}, p_g) = \sup_{\lVert f\rVert_L \le 1} > \mathbb{E}_{p_{data}}[f(x)] - \mathbb{E}_{p_g}[f(x)]$, where the critic $f$ ranges > over $1$-Lipschitz functions. Unlike JSD, $W$ varies smoothly with $p_g$ even under > disjoint support, yielding a usable gradient everywhere; the Lipschitz constraint > is enforced by weight clipping, a gradient penalty, or spectral normalization. | Cure | Acts on | Idea | | --- | --- | --- | | Feature matching | generator loss | match statistics of an intermediate $D$ layer instead of its output, broadening coverage | | Minibatch discrimination | discriminator | let $D$ see whole batches, so it can punish low diversity directly | | Wasserstein (WGAN) | value function | swap JSD for earth-mover distance — gradients survive disjoint support | | Spectral normalization | discriminator weights | bound the Lipschitz constant of $D$, stabilizing the game | | Two time-scale (TTUR) | learning rates | give $D$ and $G$ separate rates so the dynamics converge | ## Latent structure Because $G$ is a smooth, deterministic map from a low-dimensional noise space, the geometry of $\mathcal{Z}$ becomes a geometry of generated samples. Interpolating along a straight line between two noise vectors $z_0$ and $z_1$ produces a _continuous morph_ between $G(z_0)$ and $G(z_1)$, evidence that the generator has learned a meaningful manifold rather than memorizing examples.[^chollet-gan] $$ % caption: Latent interpolation: decoding points along $z_t = (1-t)\,z_0 + t\,z_1$ with $G$ morphs smoothly from $G(z_0)$ to $G(z_1)$, the mark of a continuous manifold. \begin{tikzpicture}[>=stealth, font=\footnotesize, zt/.style={circle, draw, minimum size=6mm, inner sep=0pt, font=\scriptsize}, samp/.style={draw, minimum width=12mm, minimum height=12mm}] \definecolor{acc}{HTML}{2348F2} % latent line z_0 .. z_1 \node[zt, draw=acc, text=acc] (z0) at (0,2.2) {$z_0$}; \node[zt] (za) at (2.2,2.2) {}; \node[zt] (zb) at (4.4,2.2) {}; \node[zt] (zc) at (6.6,2.2) {}; \node[zt, draw=acc, text=acc] (z1) at (8.8,2.2) {$z_1$}; \draw[acc, thick] (z0) -- (za) -- (zb) -- (zc) -- (z1); \node[font=\footnotesize, text=acc, anchor=south] at (4.4,2.55) {\texttt{latent} line from $z_0$ to $z_1$}; % generated samples below, with G arrows \node[samp] (s0) at (0,0) {}; \node[samp] (sa) at (2.2,0) {}; \node[samp] (sb) at (4.4,0) {}; \node[samp] (sc) at (6.6,0) {}; \node[samp] (s1) at (8.8,0) {}; \foreach \z/\s in {z0/s0, za/sa, zb/sb, zc/sc, z1/s1} \draw[->, black, thick] (\z) -- (\s); \node[font=\scriptsize, text=black] at (-1.0,1.1) {$G$}; % a simple morphing glyph inside each sample: a tilting bar \draw[acc, very thick] (-0.35,-0.35) -- (0.35,0.35); \draw[acc, very thick] (1.95,-0.3) -- (2.45,0.35); \draw[acc, very thick] (4.4,-0.4) -- (4.4,0.4); \draw[acc, very thick] (6.35,0.35) -- (6.85,-0.3); \draw[acc, very thick] (8.45,0.35) -- (9.15,-0.35); \node[font=\scriptsize, anchor=north] at (4.4,-0.85) {$G(z_t)$ morphs smoothly}; \end{tikzpicture} $$ ## The GAN decade Goodfellow's 2014 paper launched a fast-moving line of work, and a few landmarks explain why GANs dominated image synthesis until diffusion arrived.[^gan-orig] **Architecture came first.** The original GAN used fully connected nets on small images; the **DCGAN** established the convolutional recipe — strided convolutions, batch normalization, no pooling — that made GAN training reproducible and is the template Chollet's practical chapter follows.[^dcgan] Scaling then came from **progressive growing** and **StyleGAN**, whose style-based generator gave the disentangled, editable latent space behind the photorealistic faces that made GANs famous.[^stylegan] **Theory chased the instability.** The Wasserstein GAN reframed the objective around earth-mover distance so gradients survive disjoint support, and **WGAN-GP** replaced weight clipping with a gradient penalty to enforce the Lipschitz constraint cleanly.[^wgan] **Spectral normalization** bounded the discriminator's Lipschitz constant a different way, and the **two time-scale update rule** with the **Fréchet Inception Distance** gave the field both a stable training scheme and the sample-quality metric it had been missing.[^ttur] **Conditioning made GANs practical.** **Conditional GANs** feed a label to both networks; **pix2pix** and **CycleGAN** turned the framework into image-to-image translation, learning to map sketches to photos or horses to zebras — CycleGAN doing so without paired data, using a cycle-consistency loss in place of the pairing.[^pix2pix] These are the applications, not the theory, but they are why GANs left the lab. The learned-critic idea outlived GANs' reign as the top image model: it survives as the perceptual and adversarial losses inside super-resolution networks, as the discriminator that sharpens VAE and diffusion decoders, and as the training signal in the codec models that compress the latents diffusion runs on. ## Where GANs sit among generative models GANs are one of four major families of deep generative model, and the four trade off along three axes: whether they offer a tractable likelihood, how good their samples look, and how stable they are to train. Each [sibling lesson](/deep-learning/generative-models/autoencoders) develops one column. | Model | Tractable likelihood? | Sample quality | Training stability | Idea | | --- | --- | --- | --- | --- | | [VAE](/deep-learning/generative-models/variational-autoencoders) | lower bound (ELBO) | moderate (often blurry) | stable | maximize a variational bound | | GAN | no | high (sharp) | unstable (adversarial) | fool a learned critic | | [Autoregressive](/deep-learning/generative-models/autoregressive-and-normalizing-flows) | exact | high | stable | factorize $p(x) = \prod_t p(x_t \mid x_{