Latent Diffusion Models (LDM)

Run DDPM (topic 187) on full-resolution pixels and watch what the early steps actually do.

At t near maximum, the image is pure static. The first handful of denoising steps only remove high-frequency detail — the finest grain of the noise, the flicker at pixel-to-pixel scale. Then you look at the output and notice:

So pixel-space diffusion spends a huge fraction of its enormous compute (recall: ~1000 network calls over 786,432 numbers each) on perceptually irrelevant work. It is like cleaning a whole stadium one seat at a time when the cameras only film the field.

The Latent Diffusion Models paper (Rombach et al., 2021 / CVPR 2022) turns that observation into a design rule:

LDM is the framework that does this — and it is precisely the machinery Stable Diffusion (topic 188) ships.

LDM splits the job into two stages, trained separately:

1. Perception/compression stage: an autoencoder with encoder f and decoder d maps x → z = f(x) and back, regularized so that d(f(x)) ≈ x perceptually — the reconstruction can lose invisible detail but must keep everything visible. Diffusion then operates on z, never on x.

2. Generation stage: a diffusion model learns to denoise in latent space, and a generic conditioning mechanism steers it toward text, sketches, masks, or whatever else you feed in.

An autoencoder in one plain sentence: a network squashes an input into a small code and another network stretches the code back, and you train both so the round-trip survives.

Because the latent is spatially reduced — commonly a downsample factor f = 8, giving a 64x64x4 code for a 512x512 image — while preserving perceptual quality, the U-Net processes about 48x fewer elements per denoising step for the same visual fidelity.

The division of labor is the whole insight:

code
Copy
autoencoder  →  decides WHAT is worth keeping (perception)
diffusion    →  decides HOW to create it (generation)

Generation never has to model invisible pixel jitter, because the autoencoder already threw it away.

Training the compressor is its own small engineering story. The autoencoder is trained with a weighted sum of three losses:

• L1 pixel loss: the rebuilt image should match the original value-by-value. Simple, but alone it makes blurry outputs.
• LPIPS perceptual loss: compare deep features of the two images instead of raw pixels — "do they look the same to a trained vision model?" This is the "perceptual" in perceptual compression.
• Adversarial patch-discriminator loss: a small GAN critic looks at random patches and punishes anything that does not look like a real photo patch. This is what restores sharpness (GANs, topic 184, alive inside a diffusion pipeline).

That trio is what lets an 8x-downsampled 4-channel latent still reconstruct sharp, photoreal pixels.

Do the arithmetic on one image:

code
Copy
pixels:  512 × 512 × 3  =  786,432 numbers
latent:   64 ×  64 × 4  =   16,384 numbers
ratio:    786,432 / 16,384  ≈  48x  fewer

Every diffusion step — and there are up to 1000 of them — now touches 16k numbers instead of 786k. The decoder d runs exactly once, at the very end, to turn the finished latent back into the finished image.

The paper's second contribution is modality-agnostic conditioning: the framework does not care what you want to condition on.

Any signal y — text, a semantic layout map, a sketch, a corrupted image for super-resolution, an inpainting mask — is mapped by a domain-specific encoder tau_theta(y) into a sequence of vectors. Those vectors are injected into the U-Net via cross-attention layers, letting the denoiser "look at" the condition at every resolution block, at every step.

Cross-attention in one plain sentence: each position in one sequence (here, a latent patch) asks which parts of another sequence (the condition tokens) are relevant to me, and copies information from them — the transformer equivalent of glancing at your reference sheet mid-answer.

For text, tau_theta is a pre-trained CLIP text transformer (topic 190) — chosen because its embedding space already aligns images and language, so "red fox" lands near pictures of red foxes.

Training also applies classifier-free guidance (CFG) (topic 191): the model learns both conditional and unconditional (y = empty) denoising by randomly dropping the condition, so at inference a guidance scale w can amplify the prompt signal.

This "encoder → cross-attention" pattern is why LDM generalizes beyond text: swap tau_theta and you get ControlNet-style layout control, image-prompt conditioning, or noise-aware super-resolution, all within one framework.

Top view of the whole machine:

code
Copy
        STAGE 1 (train once)              STAGE 2 (the actual generator)
                                     ┌───────────────────────────────┐
  photo x ──► encoder f ──► z ──►· · │ noise ─► denoiser ε_θ(z_t,t,c)│· ·
              ▲               │      │              ▲                │
              │               └────  │  cross-attn  │  ← condition c │
              │                │     └──────────────┼────────────────┘
              │                ▼                    │ clean latent z_0
              └─────────── decoder d ◄ · · · · · · ┘   (decoded ONCE)
                                                  │
                                                  ▼
                                             photo x̂

And the compute picture, per denoising step:

code
Copy
pixel diffusion:  786,432 numbers × 1000 steps  =  slow, on a cluster
latent  diffusion: 16,384 numbers ×  steps      =  fast, on one GPU
                    ▲
                    └── same perceptual quality, because the
                        invisible detail was dropped before step 1

Everything to the right of the dotted line is "the generator". Stage 1 is just the real-estate agent that shrinks the apartment before the movers (the diffusion) ever show up.

Carry one picture through: an architect designing a house.

• Drafting directly on the building site — hauling bricks to try out a window placement — is absurd. That is pixel-space diffusion: making and unmaking 786,432 invisible-scale decisions.
• Instead, the architect works on a blueprint: a compressed drawing that keeps everything that matters (rooms, proportions, sight lines) and drops everything that does not (the grain of each brick). That is the latent z.
• The client's brief (text, a mood board, a floor-plan sketch) is pinned to the drafting table. At every single stroke, the architect glances at it. That is cross-attention with tau_theta(y).
• If the brief is vague, the architect is told: "also practice with no brief sometimes, so you can tell when a stroke is really driven by the client" — that is condition dropout for CFG, and a larger w at inference later means "lean harder on the brief".
• Only when the blueprint is finished does the builder show up and construct the real house in one pass: d(z_0). The construction worker (decoder) does the pixel-level work exactly once, cheaply, instead of a thousand times.

The blueprint is not the house; but every house worth building can be drawn from one. LDM is the framework that lets a diffusion model design blueprints.

LDM keeps the classic denoising U-Net but adds attention at multiple resolutions, conditioning it on t (how noisy the latent currently is) via adaptive group norm, and on c(y) via cross-attention.

Sampling proceeds exactly as in DDPM/DDIM (topic 187 / DDIM follow-ups) but entirely in latent space:

1. Start at z_T ~ N(0,I) — pure noise on the blueprint, not on the canvas.
2. Iteratively denoise to z_0 using CFG (each step: predict noise with and without the brief, overshoot toward the brief).
3. Decode x = d(z_0) once.

Because the heavy lifting happens in z, the same quality is reached at a fraction of pixel-diffusion compute — the reason text-to-image became feasible on single consumer GPUs.

# Stage 1 (separate): train autoencoder f, d on  L1 + LPIPS + GAN
z = f(x)                      # perceptually compressed latent
x_hat = d(z)

# Stage 2: latent diffusion conditioned on y via cross-attention
z_t = sqrt(abar_t) * f(x0) + sqrt(1-abar_t) * eps
eps_hat = eps_theta(z_t, t, tau(y))     # null out tau(y) sometimes -> CFG
z_0 = scheduler_reverse(z_T, eps_theta) # DDPM/DDIM walk in latent space
image = d(z_0)                          # decode once, at the end

The LDM template — compress with an autoencoder, diffuse in the latent, condition via cross-attention — has become the default generative architecture far beyond static images:

• Video: a temporal-attention U-Net or DiT diffuses over spatio-temporal latents (the autoencoder also compresses frames in time), powering Stable Video Diffusion and the text-to-video class of models (topic 192).
• 3D & audio: latent diffusion generates NeRF/implicit-surface representations and waveform/VAE-audio latents — again because the compression stage is what makes the generative stage tractable.
• Unified systems: modern MMDiT models still separate a tokenizer/autoencoder from a latent transformer — a direct descendant of the two-stage decomposition.

Stable Diffusion is simply the public, text-conditioned instantiation of LDM with a KL-latent f8 autoencoder and a CLIP text encoder — but LDM is the general framework that everything else borrows.