Denoising Diffusion Probabilistic Models

Contributions

• High-quality image synthesis without adversarial training: demonstrates that diffusion probabilistic models can generate images matching GAN-quality on CIFAR-10 and competitive on higher-resolution benchmarks, using a stable likelihood-based training objective.
• Simplified DDPM training objective: shows that a reweighted ELBO — predicting the noise ε added at each diffusion step rather than the mean μ_θ — significantly improves sample quality and simplifies implementation.
• Connection between diffusion models and denoising score matching: establishes that DDPM training is equivalent to learning a sequence of denoising autoencoders, connecting probabilistic modeling with score-based generative models.
• Lossless codelength and log-likelihood: achieves competitive NLL on CIFAR-10 (≤3.70 bits/dim) while also producing visually high-quality samples — a combination previous likelihood-based models struggled to achieve simultaneously.

Methodology

Forward Diffusion Process

The forward process gradually adds Gaussian noise to a data point x₀ over T steps:

q(x_t | x_{t-1}) = N(x_t; √(1-β_t) x_{t-1}, β_t I)

where β_t is a fixed variance schedule (linear from 10⁻⁴ to 0.02 in the original paper). After T=1000 steps, x_T ≈ N(0,I). A useful closed-form allows sampling x_t directly from x₀:

x_t = √(ᾱ_t) x₀ + √(1-ᾱ_t) ε,  ε ~ N(0,I)

where ᾱ_t = ∏_{s=1}^{t} (1-β_s).

Reverse Process (Learned Denoising)

The reverse process learns to undo each noise step:

p_θ(x_{t-1} | x_t) = N(x_{t-1}; μ_θ(x_t, t), Σ_θ(x_t, t))

The mean μ_θ is parameterized by a neural network. The key finding: predicting the noise ε added to x₀ (rather than μ_θ directly) yields better samples. The simplified training objective becomes:

L_simple = E_{t, x₀, ε} [ || ε - ε_θ(√ᾱ_t x₀ + √(1-ᾱ_t) ε, t) ||² ]

Neural Architecture

The denoising network ε_θ is a U-Net with:

• Residual blocks with group normalization
• Multi-head self-attention at 16×16 and 8×8 feature map resolutions
• Sinusoidal positional embeddings for the timestep t
• Skip connections between encoder and decoder paths

Sampling

Generation proceeds by iteratively sampling:

x_{t-1} = (1/√α_t)(x_t - (1-α_t)/√(1-ᾱ_t) · ε_θ(x_t, t)) + σ_t z,  z ~ N(0,I)

for t = T, T-1, …, 1. Requires T=1000 sequential neural network evaluations — the main computational cost.

Results

• CIFAR-10: FID 3.17 (unconditional), competitive with state-of-the-art GANs; IS 9.46. NLL ≤ 3.70 bits/dim.
• CelebA-HQ 256×256: FID 3.0, surpassing PGGAN (8.0) and comparable to StyleGAN (5.2).
• LSUN Bedroom 256×256: FID 4.90, competitive with most GAN variants.
• The model achieves good sample quality (low FID) AND reasonable NLL — a combination that flow models and VAEs previously could not match simultaneously.
• The ε-prediction parameterization consistently outperforms μ-prediction across all benchmarks.

Limitations

• Slow sampling: requires 1000 sequential denoising steps; each step requires a full U-Net forward pass, making generation ~1000× slower than a single-step GAN.
• No encoder: unlike VAEs, DDPM has no encoder mapping real images to latent codes.
• Fixed noise schedule: the linear β schedule is a hyperparameter; later work (cosine schedule, learned schedules) improves performance.
• Subsequent work addresses sampling speed (DDIM: deterministic sampling in 50 steps; consistency models; flow matching) and extends the framework to latent spaces (Stable Diffusion).