AI/ML Wiki

Denoising Diffusion Probabilistic Models

4 min read

Denoising Diffusion Probabilistic Models

Authors: Ho, Jain, Abbeel (UC Berkeley) Year: 2020

Denoising Diffusion Probabilistic Models (DDPM) established diffusion models as a competitive generative modeling paradigm, matching GAN-quality image synthesis with a stable, likelihood-based training objective. The core idea is to define a forward process that gradually corrupts data into pure Gaussian noise over T=1000 discrete steps, then train a neural network to reverse this process — learning to iteratively denoise, step by step, from noise back to a clean image. DDPM showed for the first time that such models could produce high-quality images with FID scores competitive with the best GANs, while maintaining the principled probabilistic foundation of likelihood-based models.

Core Idea

The forward (diffusion) process is fixed: it slowly adds Gaussian noise to a data sample x₀ according to a variance schedule β₁, …, β_T. After T steps, the data is completely corrupted to N(0,I). A crucial property is that x_t can be sampled directly from x₀ in closed form:

x_t = √(ᾱ_t) x₀ + √(1-ᾱ_t) ε,    ε ~ N(0, I)

where ᾱ_t is the product of noise-retention factors up to step t.

The reverse (generative) process is learned: a neural network ε_θ takes a noisy image x_t and timestep t, and predicts the noise ε that was added. Given this prediction, the model can estimate what x_{t-1} should be. Generation starts from x_T ~ N(0,I) and iteratively applies the denoiser T times to arrive at x₀.

The key simplification: rather than predicting the denoised mean directly, predicting the added noise ε yields substantially better samples. The training loss is simply:

L = E[|| ε - ε_θ(√ᾱ_t x₀ + √(1-ᾱ_t) ε, t) ||²]

This is a mean-squared-error loss — easy to implement and stable to train.

Key Concepts

  • Forward process: a fixed Markov chain that adds Gaussian noise step by step. The linear variance schedule (β from 0.0001 to 0.02 over 1000 steps) ensures gradual, complete corruption.
  • Reverse process: a learned Markov chain parameterized by a U-Net that denoises x_t → x_{t-1}. Each reverse step is also Gaussian: p_θ(x_{t-1}|x_t) = N(μ_θ, σ²I).
  • ε-prediction: predicting the noise ε rather than the mean μ is a reparameterization that turns out to upweight lower noise levels in the ELBO, empirically improving sample quality.
  • U-Net backbone: the denoising network is a U-Net with residual blocks, group normalization, multi-head self-attention at low resolutions, and sinusoidal timestep embeddings. This architectural choice was inherited by essentially all subsequent diffusion models.
  • ELBO connection: the training objective is a reweighted ELBO, connecting DDPM to variational inference and linking it to the denoising score matching literature.
  • Sampling speed: the main limitation — 1000 sequential forward passes through the U-Net per image. DDIM (Song et al.) later showed that a deterministic non-Markovian sampling process can reduce this to 50 steps with minimal quality loss.

Results

On CIFAR-10, DDPM achieves FID 3.17 (unconditional) and IS 9.46 — competitive with state-of-the-art GANs. On CelebA-HQ 256×256, FID 3.0 surpasses PGGAN and approaches StyleGAN. Critically, the model also achieves a negative log-likelihood of 3.70 bits/dim on CIFAR-10 — good by the standards of likelihood-based models. Previous generative models generally excelled at either sample quality (GANs) or likelihood (flows, VAEs), rarely both. DDPM was the first to convincingly achieve both simultaneously.

Significance

DDPM launched the diffusion model era in generative AI. Within two years, diffusion models displaced GANs as the dominant paradigm for image synthesis. The paper’s direct descendants include: DDIM (deterministic fast sampling), classifier guidance and classifier-free guidance (text-conditioned generation), DALL-E 2, Imagen, and Stable Diffusion (which applies diffusion in a VAE latent space for efficiency). The U-Net architecture introduced here became the standard backbone for all these systems.

The mathematical framework — a forward corruption process paired with a learned reverse process — also generalized beyond images to audio (WaveGrad, DiffWave), video (Video Diffusion Models), 3D shapes, and text. Score-based generative models (Song & Ermon 2019, 2020) provided a continuous-time theoretical framework that unified DDPMs with score matching, further cementing the mathematical foundations of the approach.

Graph View

Backlinks