Skip to main content

Diffusion Models

Getting a computer to draw an image that is not in its training set is hard because nobody knows how “plausible images” are spread across all possible pixel combinations. A denoising diffusion probabilistic model (DDPM) sidesteps the problem. It fixes a process that adds noise to an image step by step until only noise remains, then trains a network to undo that process by one small step. Generation starts from pure noise and undoes the process step by step until a new image is left.

The learned reverse process runs from noise xT on the left toward data x0 on the right; the dashed arrow denotes forward noising.Open full-size image

Check the direction before reading the formulas: this picture places noisy xT on the left and clean x0 on the right. The solid arrows show learned denoising p; the dashed arrow q points back toward more noise. Training samples noisy states; generation follows the reverse transitions.

The Forward Process Fades the Image into Noise Step by Step​

A variance schedule βt\beta_t sets how much noise each step adds: βt\beta_t is the variance of the noise added at step tt, and αt=1−βt\alpha_t=1-\beta_t. The previous image xt−1x_{t-1} is first scaled by αt\sqrt{\alpha_t}, then Gaussian noise of variance βt\beta_t is added:

q(xt∣xt−1)=N ⁣(xt;αt xt−1,(1−αt)I).q(x_t\mid x_{t-1}) = \mathcal{N}\!\left( x_t;\sqrt{\alpha_t}\,x_{t-1}, (1-\alpha_t)I \right).

Over many steps the scalings multiply. Let αˉt=∏s=1tαs\bar{\alpha}_t=\prod_{s=1}^{t}\alpha_s. Any timestep can be sampled directly from clean data:

xt=αˉt x0+1−αˉt ϵ,ϵ∼N(0,I).x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \qquad \epsilon\sim\mathcal{N}(0,I).

In words: the image at step tt is the original x0x_0 scaled by αˉt\sqrt{\bar\alpha_t}, plus random noise ϵ\epsilon scaled by 1−αˉt\sqrt{1-\bar\alpha_t}.

Take a scalar example: αˉt=0.64\bar\alpha_t=0.64, original x0=2x_0=2, and sampled noise ϵ=1\epsilon=1. The original is multiplied by 0.80.8 and the noise by 0.60.6, giving xt=0.8(2)+0.6(1)=2.2x_t=0.8(2)+0.6(1)=2.2. As tt grows, αˉt\bar\alpha_t shrinks and the original fades; by the last step TT, αˉT\bar\alpha_T is close to zero and almost only noise remains. Because training can jump straight to any tt, it never needs to simulate every earlier noising step.

The Network Learns to Guess the Noise in a Noisy Image​

Generation needs the step in the other direction, from xtx_t to xt−1x_{t-1}. DDPM writes this step as a Gaussian whose mean and covariance come from the network:

pθ(xt−1∣xt)=N ⁣(xt−1;μθ(xt,t),Σθ(xt,t)).p_{\theta}(x_{t-1}\mid x_t) = \mathcal{N}\!\left( x_{t-1};\mu_{\theta}(x_t,t), \Sigma_{\theta}(x_t,t) \right).

One common parameterization does not have the network output the mean directly; it predicts the noise that was added to x0x_0. A widely used simplified objective is

Ex0,t,ϵ[∥ϵ−ϵθ(αˉtx0+1−αˉtϵ,t)∥22].\mathbb{E}_{x_0,t,\epsilon} \left[ \left\| \epsilon-\epsilon_{\theta} \left( \sqrt{\bar{\alpha}_t}x_0 +\sqrt{1-\bar{\alpha}_t}\epsilon, t \right) \right\|_2^2 \right].

In words: draw a training image x0x_0, a timestep tt, and noise ϵ\epsilon; build the noisy image with the forward formula; the network ϵθ\epsilon_\theta sees the noisy image and tt, guesses ϵ\epsilon, and is penalized by the squared error.

Continue the example. Seeing 2.22.2, the network predicts noise 0.50.5, a squared noise error of (1−0.5)2=0.25(1-0.5)^2=0.25. Putting the prediction back into the forward formula implies the original estimate x^0=(2.2−0.6(0.5))/0.8=2.375\hat x_0=(2.2-0.6(0.5))/0.8=2.375. Predicting the actual sampled noise, 11, would recover 22 in this arithmetic.

The network, however, sees only 2.22.2, not this draw of ϵ\epsilon. x0=2.5x_0=2.5 with ϵ≈0.33\epsilon\approx0.33 also gives 0.8(2.5)+0.6(0.33)≈2.20.8(2.5)+0.6(0.33)\approx2.2. The same noisy image can come from many combinations of original and noise, and the squared error teaches the network the average of those possible noises: the conditional mean of the noise given xtx_t.

Training samples data, a timestep (commonly uniformly from 11 to TT), and noise, then performs one denoiser evaluation and a gradient update. The noise target and denoiser output have the same shape as xtx_t, and timestep conditioning tells the network which noise level it must handle.

Sampling Starts from Pure Noise and Removes Part of the Guessed Noise at Each Step​

The network has learned one thing: look at a noisy image and guess the noise in it. Generation starts from pure noise; each step removes part of the noise the network guesses, then adds a little fresh noise. Because the network has learned regularities shared by a large collection of images rather than any single image, after all TT steps (the DDPM paper uses T=1000T=1000) what remains is an image that resembles the training set without being in it.

For the discrete Gaussian DDPM in the original paper, Algorithms 1–2, take 0<βt<10<\beta_t<1 and αˉ0=1\bar\alpha_0=1. Sampling has three steps:

  1. Draw xTx_T from N(0,I)\mathcal N(0,I). The schedule must make αˉT\bar\alpha_T close to zero so the terminal distribution is approximately standard Gaussian; otherwise starting from N(0,I)\mathcal N(0,I) is a mismatch that no number of steps repairs.
  2. For t=T,…,1t=T,\ldots,1, compute the reverse mean μθ\mu_\theta from the network's noise prediction; for t>1t>1, draw fresh ξ∼N(0,I)\xi\sim\mathcal N(0,I) and compute xt−1=μθ+β~t ξx_{t-1}=\mu_\theta+\sqrt{\tilde\beta_t}\,\xi; at t=1t=1, return the mean without fresh noise.
  3. Return the final x0x_0.

With an epsilon-predicting network, the reverse mean is

μθ(xt,t)=1αt(xt−βt1−αˉtϵθ(xt,t)).\mu_\theta(x_t,t)=\frac{1}{\sqrt{\alpha_t}} \left(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta(x_t,t)\right).

It says: subtract part of the predicted noise from the current image xtx_t, then scale the result up by 1/αt1/\sqrt{\alpha_t}. One fixed choice for the variance of the fresh noise is β~t=βt(1−αˉt−1)/(1−αˉt)\tilde\beta_t=\beta_t(1-\bar\alpha_{t-1})/(1-\bar\alpha_t).

Staying with the example, suppose this step has βt=0.02\beta_t=0.02, so αt=0.98\alpha_t=0.98 and αˉt−1=0.64/0.98≈0.6531\bar\alpha_{t-1}=0.64/0.98\approx0.6531. The network predicts noise 0.50.5 for xt=2.2x_t=2.2:

  1. The noise removed is 0.02/0.6×0.5≈0.01670.02/0.6\times0.5\approx0.0167, leaving 2.2−0.0167≈2.18332.2-0.0167\approx2.1833.
  2. Scaling up gives 2.1833/0.98≈2.1833/0.9899≈2.20552.1833/\sqrt{0.98}\approx2.1833/0.9899\approx2.2055, which is μθ\mu_\theta.
  3. The fresh noise has variance β~t=0.02×(1−0.6531)/(1−0.64)≈0.0193\tilde\beta_t=0.02\times(1-0.6531)/(1-0.64)\approx0.0193, a standard deviation of about 0.13880.1388.

So this step gives xt−1=2.2055+0.1388 ξx_{t-1}=2.2055+0.1388\,\xi. Generation has no clean target; it repeats this with fixed weights, evaluating the denoiser again at each step. The exact forward posterior conditioned on both xtx_t and x0x_0 is Gaussian; without known x0x_0 the true reverse conditional generally is not, and DDPM approximates it with a learned Gaussian transition.

Conditioning Tells the Denoiser What to Generate​

Conditioning supplies additional information—such as a class, text representation, or measurement—to the denoiser. Guidance strengthens the effect of that condition. The widely used classifier-free guidance trains one network as both a conditional and an unconditional denoiser by replacing the condition with a null token at some probability during training. At sampling time it makes two noise predictions for the same image, with and without the condition, and amplifies their difference, so the result follows the condition more closely.

Design Choices Are Independent, and Evaluation Needs Several Measures​

  • The noise schedule, prediction target, model architecture, and sampler are separate design choices. “Predict noise with MSE” plus the reverse step above is one combination of them, not the complete definition of every diffusion model.
  • Standard sampling is iterative and calls the model many times; faster samplers trade approximation for computation.
  • Stronger guidance follows the condition more closely and also moves the trade-off between diversity and fidelity.
  • A low denoising loss does not establish perceptual quality, diversity, or usefulness, and conditional generation can reproduce biases or memorized samples from the training data.

The canonical starting point is Denoising Diffusion Probabilistic Models.

Diffusion Explainer follows preset text prompts through Stable Diffusion’s text encoder, iterative denoising, and image decoder. Step through the process and compare prompts to locate where conditioning enters. It shows a latent-diffusion application, whereas the formulas in this note describe pixel-space DDPM; the autoencoder that compresses and restores images in latent diffusion is covered in Variational Autoencoders.

Explore connectionsOpen network