Skip to main content

Variational Autoencoders

A plain autoencoder can squeeze a 28×28 handwritten digit (784 pixel values) into 2 numbers and rebuild the image from them: the encoder compresses, the decoder reconstructs. It cannot generate new digits, though. Pick an arbitrary point on the plane of those two numbers, hand it to the decoder, and the result often looks like no digit at all, because training only asked the encoded points to reconstruct well; nothing governed the empty ground between them.

A variational autoencoder (VAE) changes two things. The encoder outputs a small cloud of probability instead of a single point on the plane, and training pulls every cloud toward a standard normal distribution so that they gather around the origin and overlap. The plane is left without holes, and any point drawn from the standard normal decodes to a plausible digit. Figure 4 of the paper shows such a plane:

Digits generated across a two-dimensional latent-space grid change shape smoothly between neighboring positions.Open full-size image

Each tile decodes one latent position. Follow a row or column to see digit shapes change. The paper obtains coordinates through Gaussian inverse-CDF mapping, so this is a scan through a learned latent space, not a gallery of random independent samples.

A VAE Describes How an Image Is Generated from a Latent Variable​

Those two numbers are the latent variable zz: unseen in the data, yet deciding which image is produced. A VAE first gives zz a prior p(z)p(z), commonly standard normal, then uses the decoder to give a likelihood pθ(x∣z)p_{\theta}(x\mid z), how likely zz is to produce image xx. The probability of an image weights each zz's chance of producing it by the prior and sums (integrates) over all zz:

pθ(x)=∫p(z) pθ(x∣z) dz.p_{\theta}(x) = \int p(z)\,p_{\theta}(x\mid z)\,dz.

Now ask the reverse: given an image xx, which values of zz could explain it under the model? The answer is the posterior pθ(z∣x)p_{\theta}(z\mid x). Because the integral above cannot be computed, the posterior is also difficult to compute directly, so an encoder qϕ(z∣x)q_{\phi}(z\mid x) approximates it; the “small cloud” from the opening is qϕ(z∣x)q_{\phi}(z\mid x).

All observations share one encoder with parameters ϕ\phi. Instead of solving a separate optimization problem for each image, the network gives an image's approximate posterior in one forward pass. This reuse across observations is called amortized inference.

Training Maximizes the Evidence Lower Bound (ELBO)​

Training wants the model to give real images high probability, that is, to raise log⁡pθ(x)\log p_{\theta}(x), but that quantity contains the intractable integral. A VAE instead maximizes a lower bound on it, the evidence lower bound (ELBO).

The bound uses the KL divergence, which measures how far apart two distributions are: KL(q∥p)=Eq[log⁡q−log⁡p]\mathrm{KL}(q\|p)=E_q[\log q-\log p]. KL is nonnegative and asymmetric, and zero only when the distributions agree almost everywhere. The ELBO is

L(x;θ,ϕ)=Eqϕ(z∣x)[log⁡pθ(x∣z)]−KL ⁣(qϕ(z∣x) ∥ p(z)).\mathcal{L}(x;\theta,\phi) = \mathbb{E}_{q_{\phi}(z\mid x)} \left[\log p_{\theta}(x\mid z)\right] - \mathrm{KL}\!\left(q_{\phi}(z\mid x)\,\|\,p(z)\right).

The first term draws zz from the encoder's cloud and scores the decoder's log-likelihood of reproducing xx, rewarding a decoder that explains the observation. The second term penalizes an encoding distribution far from the prior, pulling the cloud toward the standard normal so that zz drawn from the prior is more likely to land in regions the decoder has learned. These are the two changes described at the start.

It is called a lower bound because the log-likelihood equals the ELBO plus a nonnegative KL term:

log⁡pθ(x)=L(x;θ,ϕ)+KL ⁣(qϕ(z∣x) ∥ pθ(z∣x)).\log p_{\theta}(x) = \mathcal{L}(x;\theta,\phi) + \mathrm{KL}\!\left( q_{\phi}(z\mid x) \,\|\, p_{\theta}(z\mid x) \right).

Substituting Bayes' rule pθ(z∣x)=pθ(x∣z)p(z)/pθ(x)p_\theta(z\mid x)=p_\theta(x\mid z)p(z)/p_\theta(x) into the KL definition yields this identity, provided the relevant densities and expectations are well-defined. This is the variational argument in Auto-Encoding Variational Bayes.

The extra KL on the right measures how far the encoder is from the true posterior: the better the encoder's estimate, the closer the ELBO comes to the true log⁡p(x)\log p(x); good reconstruction with a poorly estimated encoding distribution still leaves a gap between the bound and the true value.

Calling the expected log-likelihood merely “reconstruction loss” can hide an important modeling choice: Bernoulli, Gaussian, categorical, and other likelihoods imply different objectives and data assumptions, which directly affect the model's behavior.

Reparameterization Makes the Sampling Step Differentiable​

Drawing zz from the encoding distribution is a random step, and a gradient cannot pass through a draw. For a diagonal Gaussian encoder

qϕ(z∣x)=N ⁣(z;μϕ(x),diag⁡(σϕ2(x))),q_{\phi}(z\mid x) = \mathcal{N}\!\left(z;\mu_{\phi}(x), \operatorname{diag}(\sigma_{\phi}^2(x))\right),

first draw parameter-free noise, then shift and scale it:

ϵ∼N(0,I),z=μϕ(x)+σϕ(x)⊙ϵ.\epsilon\sim\mathcal{N}(0,I), \qquad z=\mu_{\phi}(x)+\sigma_{\phi}(x)\odot\epsilon.

The randomness stays in ϵ\epsilon, zz becomes an ordinary function of μϕ\mu_{\phi} and σϕ\sigma_{\phi}, and gradients flow back through them.

For a standard normal prior, the KL term has a closed form:

KL(q∥p)=−12∑j(1+log⁡σj2−μj2−σj2).\mathrm{KL}(q\|p) = -\frac{1}{2}\sum_j \left(1+\log\sigma_j^2-\mu_j^2-\sigma_j^2\right).

Each latent dimension's KL depends only on that dimension's mean and variance: the farther the mean is from 0 and the variance from 1, the larger the KL; it is zero at μj=0\mu_j=0, σj=1\sigma_j=1.

One Observation, One Training Loss​

Take a one-dimensional encoder with μ=1\mu=1, σ=0.5\sigma=0.5 and a fixed noise draw ϵ=0.2\epsilon=0.2, giving z=1+0.5×0.2=1.1z=1+0.5\times0.2=1.1. Suppose the decoder is p(x∣z)=N(z,1)p(x\mid z)=\mathcal N(z,1) and the observed value is x=2x=2. Its negative log-likelihood at this draw is (2−1.1)2/2+log⁡(2π)/2≈0.405+0.9189≈1.3239(2-1.1)^2/2+\log(2\pi)/2\approx0.405+0.9189\approx1.3239.

For the KL term, reorder the closed form above; in one dimension it is KL=(μ2+σ2−1−log⁡σ2)/2\mathrm{KL}=(\mu^2+\sigma^2-1-\log\sigma^2)/2. Substituting μ2=1\mu^2=1, σ2=0.25\sigma^2=0.25, and log⁡0.25≈−1.3863\log0.25\approx-1.3863 gives (1+0.25−1+1.3863)/2≈0.8181(1+0.25-1+1.3863)/2\approx0.8181. Adding the two terms, this one-sample negative-ELBO estimate is 2.14212.1421. Another ϵ\epsilon gives another number; training averages over many draws, and that average is what estimates the negative ELBO.

For a batch of BB observations and latent width dzd_z, the encoder emits mean and log-variance arrays of shape [B, d_z]. Set σ=exp⁡(logvar/2)\sigma=\exp(\text{logvar}/2), sample noise of the same shape, sum reconstruction log-likelihood over observation dimensions and KL over latent dimensions, then average over the batch. Averaging only the reconstruction dimensions instead of summing silently changes its weight relative to KL. Generation uses a prior draw and the decoder without the encoder; returning the decoder mean is a deterministic summary, not a full likelihood sample.

Generation and Variants​

  • Generation: sample z∼p(z)z\sim p(z), then sample or decode x∼pθ(x∣z)x\sim p_{\theta}(x\mid z).
  • Conditional VAE: condition the encoder and decoder on additional information yy.
  • β\beta-VAE: weight the KL term by β\beta to alter the rate–distortion trade-off. A larger β\beta stores less information about the input in zz (lower rate) and blurs reconstructions (higher distortion), but keeps the encoding distribution closer to the prior, so points drawn from the prior are more likely to land where the decoder has been trained.
  • Latent diffusion: latent diffusion models run a diffusion model “in the latent space of powerful pretrained autoencoders.” The encoder first compresses the image into latents, denoising happens only on those latents, and the decoder turns the result back into pixels; that encoder–decoder pair does the same compression and reconstruction as the one on this page.

When a VAE Fails​

  • Posterior collapse: a strong decoder may ignore zz, leaving qϕ(z∣x)q_{\phi}(z\mid x) close to the prior. The KL term is then zero and zz carries no information about the image.
  • Likelihood mismatch: a convenient decoder distribution can produce undesirable sample or reconstruction behavior.
  • Single draws: the negative ELBO is an expectation; a particular Monte Carlo draw is not itself guaranteed to bound log⁡p(x)\log p(x).

The canonical source is Auto-Encoding Variational Bayes.

Explore connectionsOpen network