Variational Autoencoders

A variational autoencoder (VAE) is a latent-variable generative model. It specifies a prior $p(z)$ and a decoder likelihood $p_{\theta}(x\mid z)$:

$$p_{\theta}(x) = \int p(z)\,p_{\theta}(x\mid z)\,dz.$$

The true posterior $p_{\theta}(z\mid x)$ is usually difficult to compute. An encoder $q_{\phi}(z\mid x)$ amortizes an approximate posterior across observations.

Evidence Lower Bound

Training maximizes the evidence lower bound (ELBO):

$$\mathcal{L}(x;\theta,\phi) = \mathbb{E}_{q_{\phi}(z\mid x)} \left[\log p_{\theta}(x\mid z)\right] - \mathrm{KL}\!\left(q_{\phi}(z\mid x)\,\|\,p(z)\right).$$

It is a lower bound because

$$\log p_{\theta}(x) = \mathcal{L}(x;\theta,\phi) + \mathrm{KL}\!\left( q_{\phi}(z\mid x) \,\|\, p_{\theta}(z\mid x) \right).$$

The first term rewards a decoder likelihood that explains the observation. The second keeps the approximate posterior near the prior so latent samples can support generation. Calling the first term merely “reconstruction loss” can hide an important modeling choice: Bernoulli, Gaussian, categorical, and other likelihoods imply different objectives and data assumptions.

Reparameterized Gradients

For a diagonal Gaussian encoder,

$$q_{\phi}(z\mid x) = \mathcal{N}\!\left(z;\mu_{\phi}(x), \operatorname{diag}(\sigma_{\phi}^2(x))\right),$$

sample through parameter-free noise:

$$\epsilon\sim\mathcal{N}(0,I), \qquad z=\mu_{\phi}(x)+\sigma_{\phi}(x)\odot\epsilon.$$

This moves randomness outside the differentiable path through $\mu_{\phi}$ and $\sigma_{\phi}$.

For a standard normal prior, the KL term has a closed form:

$$\mathrm{KL}(q\|p) = -\frac{1}{2}\sum_j \left(1+\log\sigma_j^2-\mu_j^2-\sigma_j^2\right).$$

Generation and Variants

• Generation: sample $z\sim p(z)$, then sample or decode $x\sim p_{\theta}(x\mid z)$.
• Conditional VAE: condition the encoder and decoder on additional information $y$.
• $\beta$-VAE: weight the KL term by $\beta$ to alter the rate–distortion trade-off. A larger weight does not guarantee a uniquely disentangled or semantically meaningful representation.

Failure Modes

• Posterior collapse: a strong decoder may ignore $z$, leaving $q_{\phi}(z\mid x)$ close to the prior.
• Approximation gap: the chosen variational family may poorly represent the true posterior.
• Likelihood mismatch: a convenient decoder distribution can produce undesirable sample or reconstruction behavior.
• Latent interpretation: smooth interpolation does not prove that latent coordinates correspond to independent human concepts.
• Anomaly detection: reconstruction or likelihood scores can fail on unfamiliar data and need task-specific validation.

The canonical source is Auto-Encoding Variational Bayes. Framework training code is intentionally omitted because the model assumptions and ELBO are the durable part.