AI/ML Wiki

Auto-Encoding Variational Bayes

4 min read

Auto-Encoding Variational Bayes

Authors: Kingma, Welling Year: 2013

The Variational Autoencoder (VAE) is one of the foundational papers of modern deep generative modeling. It marries deep neural networks with Bayesian latent variable models, introducing an elegant probabilistic framework in which an encoder network maps observed data to a distribution over a compact latent space, and a decoder network maps samples from that latent space back to the data space. The two networks are trained jointly by maximizing a tractable lower bound on the data log-likelihood. The key technical innovation — the reparameterization trick — makes it possible to backpropagate through a stochastic sampling step, enabling end-to-end gradient-based optimization. VAEs produced the first generative models with smooth, semantically organized latent spaces that could be sampled and interpolated.

Core Idea

The VAE is a latent variable model p_θ(x) = ∫ p_θ(x|z) p(z) dz where x is the observed data and z is a continuous latent variable with prior p(z) = N(0, I). The integral is intractable, so the model is trained by instead maximizing the Evidence Lower BOund (ELBO):

L = E_{q_φ(z|x)}[log p_θ(x|z)] - KL(q_φ(z|x) || p(z))

Here q_φ(z|x) is an encoder neural network that outputs Gaussian parameters (μ, σ²) for the approximate posterior. The first term measures reconstruction quality; the second is a KL divergence that regularizes the latent space toward the prior.

The reparameterization trick makes gradient computation possible: instead of sampling z ~ N(μ, σ²) directly, the model writes z = μ + σ·ε where ε ~ N(0,I) is a fixed noise sample. Gradients then flow through μ and σ unobstructed.

Key Concepts

  • Encoder (recognition network): a neural network q_φ(z|x) mapping each input x to a distribution over z — specifically, predicting the mean μ and log-variance log σ² of a diagonal Gaussian.
  • Decoder (generative network): a neural network p_θ(x|z) mapping a latent code z to a distribution over x (Gaussian for continuous data, Bernoulli for binary data).
  • ELBO: the variational lower bound on log p(x). Maximizing the ELBO simultaneously improves reconstruction (log-likelihood term) and regularizes the latent space (KL term).
  • KL divergence term: for Gaussian q and p, this has a closed form: -½ Σ(1 + log σ² - μ² - σ²). It penalizes latent codes that deviate far from the standard normal prior, enforcing continuity of the latent space.
  • Reparameterization trick: rewrites the stochastic z as a deterministic function of the parameters plus independent noise, enabling backpropagation through sampling.
  • Amortized inference: a single encoder network computes posterior parameters for all data points, rather than optimizing variational parameters per data point (as in mean-field VI).
  • Latent interpolation: because the latent space is continuous and regularized toward a smooth Gaussian, interpolating between two latent codes produces semantically meaningful intermediate outputs.

Results

On MNIST, a 2D VAE latent space organizes digit classes into distinct clusters without class supervision — evidence that the model discovers meaningful structure. Quantitative log-likelihood (−86.47 nats) outperforms the wake-sleep algorithm (−89.0 nats). On the Frey Face dataset, a 2D latent space captures pose and expression as interpretable axes. Latent space interpolation produces smooth transitions between face identities and poses.

Significance

The VAE established the blueprint for deep probabilistic generative models with structured latent spaces. Its influence spans three main directions. First, it introduced the ELBO and reparameterization trick as standard tools for training deep latent variable models — both remain central to variational inference literature. Second, it motivated hierarchical VAEs (NVAE, VD-VAE) and flow-augmented posteriors (IAF, normalizing flows) that improve posterior expressiveness and generation quality. Third, and most consequentially for modern AI, the VAE’s continuous latent space concept directly influenced diffusion models and latent diffusion models (Stable Diffusion encodes images into a VAE latent space before applying the diffusion process, dramatically reducing computational cost).

VAEs complement GANs and diffusion models in the generative modeling landscape: they are computationally efficient, provide explicit latent encodings of inputs (unlike GANs), and offer a principled probabilistic training objective (unlike many GAN variants). They remain the standard approach when interpretable or controllable latent representations are needed.

Graph View

Backlinks