Variational Autoencoders
A plain autoencoder can squeeze a 28×28 handwritten digit (784 pixel values) into 2 numbers and rebuild the image from them: the encoder compresses, the decoder reconstructs. It cannot generate new digits, though. Pick an arbitrary point on the plane of those two numbers, hand it to the decoder, and the result often looks like no digit at all, because training only asked the encoded points to reconstruct well; nothing governed the empty ground between them.
A variational autoencoder (VAE) changes two things. The encoder outputs a small cloud of probability instead of a single point on the plane, and training pulls every cloud toward a standard normal distribution so that they gather around the origin and overlap. The plane is left without holes, and any point drawn from the standard normal decodes to a plausible digit. Figure 4 of the paper shows such a plane:
Open full-size imageEach tile decodes one latent position. Follow a row or column to see digit shapes change. The paper obtains coordinates through Gaussian inverse-CDF mapping, so this is a scan through a learned latent space, not a gallery of random independent samples.
A VAE Describes How an Image Is Generated from a Latent Variable
Those two numbers are the latent variable : unseen in the data, yet deciding which image is produced. A VAE first gives a prior , commonly standard normal, then uses the decoder to give a likelihood , how likely is to produce image . The probability of an image weights each 's chance of producing it by the prior and sums (integrates) over all :
Now ask the reverse: given an image , which values of could explain it under the model? The answer is the posterior . Because the integral above cannot be computed, the posterior is also difficult to compute directly, so an encoder approximates it; the “small cloud” from the opening is .
All observations share one encoder with parameters . Instead of solving a separate optimization problem for each image, the network gives an image's approximate posterior in one forward pass. This reuse across observations is called amortized inference.
Training Maximizes the Evidence Lower Bound (ELBO)
Training wants the model to give real images high probability, that is, to raise , but that quantity contains the intractable integral. A VAE instead maximizes a lower bound on it, the evidence lower bound (ELBO).
The bound uses the KL divergence, which measures how far apart two distributions are: . KL is nonnegative and asymmetric, and zero only when the distributions agree almost everywhere. The ELBO is
The first term draws from the encoder's cloud and scores the decoder's log-likelihood of reproducing , rewarding a decoder that explains the observation. The second term penalizes an encoding distribution far from the prior, pulling the cloud toward the standard normal so that drawn from the prior is more likely to land in regions the decoder has learned. These are the two changes described at the start.
It is called a lower bound because the log-likelihood equals the ELBO plus a nonnegative KL term:
Substituting Bayes' rule into the KL definition yields this identity, provided the relevant densities and expectations are well-defined. This is the variational argument in Auto-Encoding Variational Bayes.
The extra KL on the right measures how far the encoder is from the true posterior: the better the encoder's estimate, the closer the ELBO comes to the true ; good reconstruction with a poorly estimated encoding distribution still leaves a gap between the bound and the true value.
Calling the expected log-likelihood merely “reconstruction loss” can hide an important modeling choice: Bernoulli, Gaussian, categorical, and other likelihoods imply different objectives and data assumptions, which directly affect the model's behavior.
Reparameterization Makes the Sampling Step Differentiable
Drawing from the encoding distribution is a random step, and a gradient cannot pass through a draw. For a diagonal Gaussian encoder
first draw parameter-free noise, then shift and scale it:
The randomness stays in , becomes an ordinary function of and , and gradients flow back through them.
For a standard normal prior, the KL term has a closed form:
Each latent dimension's KL depends only on that dimension's mean and variance: the farther the mean is from 0 and the variance from 1, the larger the KL; it is zero at , .
One Observation, One Training Loss
Take a one-dimensional encoder with , and a fixed noise draw , giving . Suppose the decoder is and the observed value is . Its negative log-likelihood at this draw is .
For the KL term, reorder the closed form above; in one dimension it is . Substituting , , and gives . Adding the two terms, this one-sample negative-ELBO estimate is . Another gives another number; training averages over many draws, and that average is what estimates the negative ELBO.
For a batch of observations and latent width , the encoder emits mean and log-variance arrays of shape [B, d_z]. Set , sample noise of the same shape, sum reconstruction log-likelihood over observation dimensions and KL over latent dimensions, then average over the batch. Averaging only the reconstruction dimensions instead of summing silently changes its weight relative to KL. Generation uses a prior draw and the decoder without the encoder; returning the decoder mean is a deterministic summary, not a full likelihood sample.
Generation and Variants
- Generation: sample , then sample or decode .
- Conditional VAE: condition the encoder and decoder on additional information .
- -VAE: weight the KL term by to alter the rate–distortion trade-off. A larger stores less information about the input in (lower rate) and blurs reconstructions (higher distortion), but keeps the encoding distribution closer to the prior, so points drawn from the prior are more likely to land where the decoder has been trained.
- Latent diffusion: latent diffusion models run a diffusion model “in the latent space of powerful pretrained autoencoders.” The encoder first compresses the image into latents, denoising happens only on those latents, and the decoder turns the result back into pixels; that encoder–decoder pair does the same compression and reconstruction as the one on this page.
When a VAE Fails
- Posterior collapse: a strong decoder may ignore , leaving close to the prior. The KL term is then zero and carries no information about the image.
- Likelihood mismatch: a convenient decoder distribution can produce undesirable sample or reconstruction behavior.
- Single draws: the negative ELBO is an expectation; a particular Monte Carlo draw is not itself guaranteed to bound .
The canonical source is Auto-Encoding Variational Bayes.