Diffusion Models: The Big Picture

You have two generative models behind you now.

The GAN: one network, one forward pass, a finished image — but trained by an adversarial game that oscillates, vanishes, and mode-collapses (topic 184).

The VAE: beautifully stable training — but blurry, because its pixel-independent decoder paints the average of everything plausible (topic 185).

So the question becomes

Diffusion's answer: never try to create an image in one step.

Think about the one-shot leap the GAN asks its generator to make: random numbers → a complete face. That is a monstrous, brittle function to learn in one network call.

Instead, imagine a photograph developing out of TV static — grain by grain. Each individual step is tiny: "remove a little of the hiss, sharpen a little of the edge." A small correction is easy to learn. A thousand small corrections chained is a masterpiece — and each correction is supervised by a loss as simple as "guess the dirt I just sprinkled."

This topic is that recipe: a fixed forward destruction, a learned reverse repair, and one MSE loss running the whole show.

Diffusion models generate by iterative refinement rather than a single shot. The recipe has two halves:

1. A forward diffusion process you do not learn: over T steps, progressively add Gaussian noise to a real sample x_0 until at x_T it is statistically indistinguishable from pure noise N(0, I). Each step is a fixed Markov chain with a known noise schedule beta_t.
2. A reverse process you do learn: a neural network looks at a noisy x_t plus the timestep t and predicts how to remove a little noise, walking a sample from noise back to data.

Bold the two halves, because beginners always mix them up:

• Forward = fixed, mindless, one-way. Just math: mix a little more noise in, every step, until the data is gone. No network involved. It only has to make training pairs.
• Reverse = learned. One network does all the work, and it does only one thing: at whatever noise level it is shown, identify the noise hiding inside.

The elegance is that the intractable reverse marginals are approximated by a Gaussian whose mean the network outputs, and the entire training signal collapses to one simple regression: predict the noise you added.

Why "predict the noise" instead of "produce the image"? Because noise is easy to label: you added it yourself during the forward process, so you know the answer before the network even speaks. Training needs no humans, no critics, no games — just the sprinkled dirt as ground truth.

Use a single number for x_0: say a pixel value 1.0. A noisy sample at step t is formed by mixing:

x_t = sqrt(alpha_bar_t) · x_0 + sqrt(1 - alpha_bar_t) · epsilon

where epsilon is a standard-normal dice-roll and alpha_bar_t (shorthand: ᾱ) is how much signal survives by step t — set by the schedule beta_t. Follow along with the roll fixed at epsilon = +0.5:

• t=1: ᾱ = 0.8 → x_1 = 0.894·1.0 + 0.447·0.5 = 1.12 — still clearly "about 1", slight hiss added.
• t=2: ᾱ = 0.5 → x_2 = 0.707·1.0 + 0.707·0.5 = 0.85 — signal and noise now equal-weighted.
• t=3: ᾱ = 0.01 → x_3 ≈ 0.10 + 0.99·0.5 ≈ 0.60 — the 1.0 barely matters; this is mostly dice.

Watch the ratio: sqrt(ᾱ) shrinks toward 0, sqrt(1-ᾱ) grows toward 1. At T, whatever the data was, x_T is noise. Destruction complete — and notice: to train, you do not walk 1000 steps per image. The mix formula means you can jump straight to any step t, add that step's noise, and ask the network:

Network answers epsilon_theta(x_t, t). Loss: ||epsilon - epsilon_theta(x_t, t)||^2. That MSE is the entire training objective. Sample a random t per image per batch, done.

Then at generation time: roll pure noise at t=T, feed it in, and run the network backwards — T, T−1, … 1 — each call subtracting the noise it predicts. Out walks a fresh sample from a distribution no one ever wrote down.

The forward process is an escalator going down into fog; the learned reverse is the walk back up, one visible step at a time:

code
Copy
FORWARD (fixed, no learning):            REVERSE (the trained denoiser):
 x_0  sharp photo                          z ~ N(0,I)  pure static
  │  + a little noise                       │  "predict the noise" → subtract it
 x_1  faint hiss                            │
  │                                          ▼
  ▼                                         x_(T-1) blurry shapes
 x_500  half signal, half hiss              │  denoise again
  │                                          ▼
  ▼                                         x_100  edges, wrong fingers
 x_T  indistinguishable from TV static      │  ... 20-1000 steps later
                                            ▼
                                          x_0  a finished image

Key geometry to trust: the reverse is only tractable because each step is small. Removing 1% of the noise is an easy regression; removing 100% is the GAN's impossible one-shot leap.

And a second picture — what the network sees per training step:

code
Copy
dataset photo x_0 ──► mix with known noise ε ──► x_t ──► network(x_t, t) ──► ε-hat
                            │                                                 │
                            └──────────── loss = ||ε − ε-hat||² ─────────────┘

No label. No teacher. No adversary. The supervision is the noise you sprinkled yourself — self-labeled by construction. That is why it scales as calmly as image classification.

Carry this through: generation is restoring a film that starts as pure snow.

• The forward process is time damaging the film: each day a little more grain, until the archive copy is unrecognizable static. Nobody learns this — it is physics, and physics means you can simulate any damage level on demand (the jump formula).
• The restorer (the network) has only one skill: look at any frame at any damage level and name the grain that's on it — not paint the missing scene, just identify the dirt so it can be subtracted.
• The timestep t is the day-counter taped to each canister: the restorer must know how badly damaged the film should be, because "remove a little grain" means something different on day 50 than on day 950. (That is why the network is conditioned on t — Section 7.)
• Sampling is the restoration marathon: start from day-T total snow, restore one day's worth, tape down the next frame, repeat — 1000 careful passes to finish the reel.
• The speed problem (Section 8) is exactly the budget meeting: "must we screen every day of damage?" DDIM is the clever editor who samples every 20th day and stitches (20-50 steps); distillation is training a prodigy intern who has watched the whole marathon so often they can restore the entire film in ONE pass (1-4 steps) — occasionally missing subtle detail (diversity, fine texture) but fast enough for live products.

The 2020-2023 literature converged on one family seen through three framings:

• DDPM (Ho et al., 2020): a discrete Markov-chain / variational-bound view with a reweighted ELBO that reduces to ||epsilon - epsilon_theta(x_t, t)||^2.
• Score-based SDEs (Song et al., 2021): view the process as a continuous stochastic differential equation and train a network to estimate the score grad_x log p(x); sampling integrates the reverse-time SDE (or its deterministic probability-flow ODE).
• Flow Matching / Rectified Flow (2022-2023): learn a velocity field that transports noise to data along (near-)straight paths, removing the hand-designed noise schedule. This is the backbone of Stable Diffusion 3 and Google's Imagen-style models.

Mathematically the DDPM noise-prediction network and the score network are related by score = -epsilon_theta / sigma_t; they are the same object under a change of variables.

The film-restoration decode of the three lenses: DDPM counts damage days (discrete steps), the SDE view treats damage as continuous rot you integrate backwards, and rectified flow claims the best restoration path is a straight line from static to scene. All three train "remove what you see"; the differences are parameterization and geometry.

The canonical denoiser is a U-Net: an encoder/decoder with skip connections, multi-resolution attention, and group normalization, conditioned on t via FiLM and on text via cross-attention to a text encoder. This design powered Stable Diffusion and Imagen.

The 2023 Diffusion Transformer (DiT) from Peebles & Xie replaced the U-Net with a vanilla ViT operating on latent patches, conditioning on t and the prompt through in-context tokens and adaLN. DiT scaled like LLMs did - quality rose predictably with parameters and compute - and it is now the default backbone for state-of-the-art image (SD3/FLUX MMDiT) and video (Sora-class) models. The lesson from 2024-2026: the diffusion objective is backbone-agnostic, so transformers inherited the scaling wins.

Note what "latent" means in all of this: the denoiser does not run on 512x512 pixels but on the VAE's 8x-compressed space (previous topic) — the reason the marathon of 20-1000 network calls is affordable at all.

t = torch.randint(0, T, (B,), device=dev)
noise = torch.randn_like(x0)
x_t = sqrt(abar[t])[:,None,None,None] * x0 + sqrt(1-abar[t])[:,None,None,None] * noise
pred = model(x_t, t, text_emb)            # denoising U-Net / DiT
loss = F.mse_loss(pred, noise)            # that is the whole objective

Generation integrates the reverse process step by step, which is where diffusion pays: classic DDPM ancestral sampling needed ~1000 network evaluations. The field relentlessly attacked this:

• DDIM (Song et al., 2020): a deterministic, non-Markovian sampler that reuses the trained model and generates in 20-50 steps. Plain words: because the reverse can be viewed as solving an ODE along a fixed trajectory, you may leap along that same trajectory instead of crawling the stochastic chain — same trained epsilon_theta, far fewer calls.
• Higher-order ODE solvers (DPM-Solver, Heun, Adams) reach good quality in ~10-20 steps.
• Distillation to 1-4 steps: Progressive Distillation, Consistency Models / LCM, and adversarial distillation (SDXL-Turbo, SD3-Turbo) bake a full sampler into a single forward pass.

The trade-off is consistent: fewer steps → lower diversity and detail and more artifacts; the "fast" frontier (rectified-flow + guidance-distilled transformers) is what makes real-time diffusion viable in products today.

The scoreline, end to end:

code
Copy
~1000 (DDPM)  →  20-50 (DDIM)  →  10-20 (ODE solvers)  →  1-4 (distilled)
stable training kept intact at every step of that compression