Diffusion Models: What Denoising Is Actually Learning
Diffusion models generate images by reversing a noise process. But what is the network actually learning? Not the image — it learns the score function, the gradient of the log probability of data. This is why diffusion models surpass GANs on image quality without adversarial training. The math is simpler than it looks once you see the forward and reverse processes for what they are.
Diffusion models belong to the class of generative models that learn to turn random noise into data. Unlike GANs (which learn through adversarial play) or VAEs (which learn through an ELBO objective), diffusion models are trained with a deceptively simple objective: predict the noise that was added to data.
The forward process: destroying information gradually
Given a data sample x_0, the forward process adds Gaussian noise over T steps. At each step t: x_t = sqrt(1 - β_t) * x_{t-1} + sqrt(β_t) * ε, where β_t is a variance schedule (small values like 0.0001 to 0.02) and ε ~ N(0, I). After T=1000 steps with a well-chosen schedule, x_T is approximately pure Gaussian noise — the original data is completely destroyed. There are no learnable parameters in the forward process. It is a fixed Markov chain.
The reverse process: learning to denoise
The reverse process tries to invert the forward process: starting from x_T ~ N(0, I), iteratively denoise to recover x_0. At each step, a neural network ε_θ(x_t, t) predicts the noise that was added at step t. The training objective is simple MSE: L = E[||ε - ε_θ(x_t, t)||^2]. The model predicts noise; you subtract it to get a cleaner estimate; repeat for all T steps.
What the network is actually learning: the score function
Predicting noise is equivalent to estimating the score function: ∇_{x_t} log p(x_t). The score is the gradient of the log probability density — it points from low-probability regions toward high-probability regions. A perfectly trained model learns to point toward the data manifold from any noise level. Sampling is then following this gradient field: start from noise, repeatedly step toward higher density. This is denoising score matching, the statistical framework underlying diffusion models.
Why diffusion models beat GANs
GANs have high sample quality but three chronic problems: mode collapse (the generator covers only some modes of the data distribution), training instability (the discriminator and generator can diverge), and limited diversity (evaluating FID rewards quality but not full coverage). Diffusion models have no adversarial training. The score function is learned from the full training distribution. They cover all modes and produce diverse, high-quality samples. The trade-off is slow sampling: T=1000 denoising steps per image. DDIM and consistency models reduce this to 10-50 steps without quality loss.
The architecture: U-Net with attention
DDPM (Ho et al., 2020) uses a U-Net as the denoising network. U-Net is a convolutional architecture with skip connections between encoder and decoder stages — originally designed for medical image segmentation, it is ideal for the noise-prediction task because it operates at the image resolution while capturing multi-scale context. Attention layers are added at the lower spatial resolutions. The timestep t is injected as a conditioning signal (sinusoidal embedding, like positional encoding in Transformers) — the network must denoise differently depending on how much noise is present.
Conditional generation: classifier-free guidance
Unconditional diffusion models sample from the full data distribution. Text-to-image (Stable Diffusion, DALL·E 2) conditions generation on a text prompt. Classifier-free guidance (Ho & Salimans, 2021) trains the model jointly on conditional and unconditional denoising. At sampling time, the score is interpolated: ε_guided = ε_uncond + w * (ε_cond - ε_uncond). The guidance weight w controls the trade-off between diversity (low w) and prompt fidelity (high w). Values of w=7.5 are common — enough to steer generation toward the prompt while preserving image quality.
Latent diffusion: scaling to high resolution
Running diffusion in pixel space at 512×512 requires denoising a 786,432-dimensional vector at each step. Latent diffusion (Rombach et al., 2022) instead trains the diffusion model in the compressed latent space of a VAE. A VAE encodes 512×512 images to 64×64×4 latents — a 48× compression. Denoising is done in this small latent space; the VAE decoder converts the final latent to a pixel image. This is what makes Stable Diffusion practical on a single GPU.
Try on Colab: run DDPM on MNIST. Train the U-Net noise predictor for 10 epochs. Generate samples by running 1000 denoising steps from pure noise. Then implement DDIM sampling (20 steps) on the same trained model and compare sample quality. The speed improvement from 1000 to 20 steps with minimal quality loss demonstrates why deterministic samplers replaced stochastic ones.