Generative Adversarial Networks (GANs)

Suppose you want a program that invents photorealistic faces.

The textbook plan for "learning data" is: write a formula for the probability of every possible image, then maximize how likely the training set is under it (maximum likelihood).

Sounds fine until you count. An image of 1024×1024 RGB has 3 million numbers describing it. The space of all such arrays is astronomically huge; exactly a dust-mote fraction are real faces. Nobody can write down that probability function.

So the question becomes

That is the move Goodfellow et al. (2014) made. Skip the formula. Hire two neural networks, make them compete, and let the competition itself define "realistic".

No explicit likelihood is ever computed. The generator learns purely by trying not to get caught.

As a bonus, this framing gives sharp images (you are graded on fooling a critic, not on averaging probabilities — remember that contrast when we meet VAEs next topic). The cost: a training game that is famously hard to keep balanced. This topic is the game, the balance, and what broke.

A GAN is simply

Meet the players:

• The generator G(z; theta) — the forger. It takes a vector of pure random noise z ~ p_z (usually standard normal — a handful of random decimals, like [0.31, -1.2, 0.04, …]) and maps it to one full synthetic sample in data space. Feed it noise, out comes a face.
• The detective D(x; phi) — a classifier that takes any image (real or forged) and outputs a scalar probability: how likely is this from the real data rather than the forger?

Both networks are trained simultaneously against a single value function — the minimax game (one player pushes it up, the other pushes it down):

min_G max_D  V(D, G) = E_{x~p_data}[ log D(x) ] + E_{z~p_z}[ log(1 - D(G(z))) ]

Read the two terms like a rulebook:

• log D(x) over real images: detective wants these scores HIGH (D(x) → 1).
• log(1 − D(G(z))) over forgeries: detective wants D(G(z)) LOW (so 1−D → 1).
• The forger plays the same board in reverse: it wants D(G(z)) HIGH — caught? push the knob the other way.

The game only works through gradients: D's verdict is differentiated back through D into G (D's weights just watched; G's weights learn "move the forged face in whatever direction raised p(real)").

Question the math answers: if the detective were perfect, what would the game look like?

Step 1 — the optimal detective. At any image x, D should output the chance it is real among the mixed deck it sees. With reals drawn from p_data and fakes from p_g:

D*(x) = p_data(x) / (p_data(x) + p_g(x))

Sanity check: where fakes are unknown (p_g = 0), D* = 1. Where forgeries outnumber real data, D* → 0. Reasonable.

Step 2 — what the forger faces against that detective. Plug D* back into the value function and the game turns into a classic distance between distributions: the generator's objective becomes minimizing the Jensen-Shannon divergence JS(p_data || p_g) — a symmetric "how different are these two distributions" measure (0 when identical, log 2 in nats when fully disjoint).

Step 3 — the scoreboard at equilibrium. The unique global minimum, J* = -2 log 2 ≈ -1.386, occurs only when p_g = p_data.

Tiny coin-toss version: imagine the images are just coins and "real" data is heads-biased 60%.

• If the forger produces fair coins (50%), a sharp detective can spot the statistical gap → scores like 0.7 vs 0.3 → game not over.
• If the forger also produces 60%-heads coins, the detective's best guess is a coin-flip: D = 0.5 everywhere. That confusion is the finish line, and E[log 0.5] = -log 2 per term is exactly the -2 log 2 above.

So "the detective gives up and guesses" is not training failure — it is the definition of victory: fakes statistically indistinguishable from data.

Picture the real-image distribution as a mountain range over pixel-space, and the generator's output as a cloud of points it can place.

code
Copy
round 1:  G sprays points anywhere          D: easy day
            ·  ·   ·                        "real: 1.0, fake: 0.02"
          ▄▄ real-data ridge ▄▄             gradient to G: move toward ridge

round 2:  G crowds the ridge                 D: sharper
            ▲▲▲▲ ▲▲ ▲▲▲                     now only tiny gaps expose fakes
          ▄▄▄▄▄ real-data ridge ▄▄▄▄        gradient shrinks as D perfects

equilibrium: G's cloud = the ridge itself    D: 0.5 everywhere
          ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄              game over = p_g = p_data

Two warnings readable straight from this picture:

• If D gets too good too fast (round 2 with a flawless detective), the gradient signal "move toward the ridge" becomes flat — nothing left to teach G. That is vanishing gradients, the game's favorite way to die.
• The cloud only needs to touch the ridge somewhere to fool a weak D. It can crowd one peak and ignore the rest of the range — that is mode collapse: great faces, all of the same face.

The loop as plumbing, since every implementation follows it:

code
Copy
z ~ N(0,I) ─► G(z) ─► fake ─┐
                             ├─► D ─► p(real) ─► gradients ─► update D and G
real x ~ p_data ────────────┘        (alternate mini-batches)

Carry this one through: a master forger versus an art detective in a courtroom.

• Early days: the forger's paintings are obviously fake. The detective condemns them instantly. The forger learns only from the detective's criticism — every verdict is a gradient.
• Mid-career: the forger studies what fools the detective and paints exactly that. The detective goes back to school (retrains on the new fakes) and comes back sharper. An arms race — the only training signal either network ever gets is the other one.
• Equilibrium: the forger produces works indistinguishable from the real collection as a distribution — same brushwork statistics, same range of subjects. The detective can do no better than "50/50, your honor". That is p_g = p_data.
• Mode collapse = the forger finds ONE painting that always passes and forges only that, forever. The detective (trained on a mix) may still catch it eventually — but diversity is dead.
• Vanishing gradients = the detective becomes so flawless that every verdict is a confident "FAKE". The forger gets no useful direction, just a wall. (The original paper's fix: early on, train the forger to maximize the detective's mistakes instead — hear "real" at least as often as possible.)
• WGAN's critic = replace the courtroom verdict (guilty/not, 0 to 1) with a price appraisal: "this piece is worth 0.4 vs the real one's 1.7" — an unbounded score that keeps giving direction even when fakes are hopeless. That reframing is the deeper fix in Section 6.

Unlike minimizing a fixed loss, a GAN seeks a Nash equilibrium of a dynamic game, and gradient descent on a game need not converge. Practical failure modes:

• Vanishing gradients: if D becomes too strong it perfectly separates real from fake, so log(1 - D(G(z))) saturates and the gradient to G collapses to zero. The original paper's remedy is to maximize log D(G(z)) early instead.
• Mode collapse: G discovers that one sample (or a few) fools D and collapses onto a single mode, sacrificing diversity.
• Non-convergence / oscillation: D and G chase each other in limit cycles without ever settling.

Stabilization tricks include label smoothing, feature matching, balancing the D/G update ratio, minibatch discrimination, and adding history/EMA of parameters. The deeper fix, however, changed the objective itself:

DCGAN (Radford, 2015) established the recipe - strided convolutions in G, convolutions with no pooling in D, batch norm, LeakyReLU - making adversarial training reproducible. cGAN (Mirza, 2014) conditions both networks on a label y, unlocking targeted generation. pix2pix and CycleGAN added adversarial losses to image-to-image translation, even without paired data.

Progressive GAN (Karras, 2017) grew resolution from 4x4 upward, stabilizing high-res training. StyleGAN (2018) injected a learned "style" vector via adaptive instance normalization at every resolution, giving unprecedented control and the famous latentspace interpolation. StyleGAN2 (2019) removed the path-length regularization that caused droplet artifacts and adopted equalized learning rates. StyleGAN3 (2021) made the generator aliasing-free so features no longer "grow" with resolution.

In the forger analogy: DCGAN set the workshop standards; cGAN gave the forger client requests ("paint me a sedan"); Progressive GAN trained at postage-stamp size before graduating to posters; StyleGAN moved the forger's dials from "copy this sketch" to "set the lighting, the age, the hair" — style control at every resolution.

Here is the WGAN-GP training step in miniature — note the critic is scored, not classified, and the penalty pushes interpolation points toward slope 1:

# critic (not a probabilistic discriminator)
for _ in range(n_critic):
    real = sample_data(B)
    z = torch.randn(B, latent_dim, 1, 1)
    fake = G(z).detach()
    eps = torch.rand(B, 1, 1, 1)
    x_hat = eps * real + (1 - eps) * fake
    grad = autograd.grad(C(x_hat), x_hat, torch.ones_like(real),
                        create_graph=True)[0]
    gp = ((grad.norm(2, dim=1) - 1) ** 2).mean()
    loss_c = C(fake).mean() - C(real).mean() + 10 * gp
    loss_c.backward(); opt_c.step(); opt_c.zero_grad()

z = torch.randn(B, latent_dim, 1, 1)
loss_g = -C(G(z)).mean()      # generator maximizes critic score
loss_g.backward(); opt_g.step(); opt_g.zero_grad()

For headline text-to-image, diffusion and rectified-flow models (Stable Diffusion 3, FLUX, Imagen 3) have largely overtaken GANs because they train stably and scale with prompts. But adversarial training is far from obsolete:

• Vocoders & audio: HiFi-GAN and WaveGAN still convert mel-spectrograms to waveforms; the adversarial discriminator is why neural TTS sounds crisp.
• Super-resolution: ESRGAN / Real-ESRGAN use adversarial perceptual losses to hallucinate realistic detail.
• One-step diffusion distillation: SDXL-Turbo and the Adversarial Diffusion Distillation (ADD) recipe add a GAN-style discriminative head so a diffusion model can produce an image in a single step. The adversarial idea survives inside the diffusion stack.
• Real-time, high-fidelity synthesis: StyleGAN-T and its successors still ship in face/asset pipelines where a single forward pass must render at interactive frame rates.

The lasting bargain of the GAN: one forward pass = one finished image. Diffusion needs tens of network evaluations; the forger needs one. When latency is the product — interactive apps, on-device faces, live video — the detective's old weapon is still in the drawer, loaded.