Diffusion Models Explained: How AI Generates Images and Video

Ask a diffusion model for an image of a fox in a library, and it starts from pure static — literally random noise, no structure at all — and repeatedly refines that static into a coherent picture over dozens of steps. That’s a strange way to generate anything, and the fact that it works at all is worth understanding, because the same core mechanism now drives most state-of-the-art image and video generation.

The Forward Process: Destroying Information on Purpose

Diffusion models are trained by running the process backwards from how you’d expect. You don’t start by teaching the model to draw; you start by teaching it to denoise, and to get denoising examples, you first have to noise things up.

Take a real image and add a small amount of Gaussian noise to it. Add a bit more. Repeat this a few hundred or a thousand times, on a fixed schedule, and eventually the image is indistinguishable from pure noise. This is the forward process, and it’s deliberately simple: it’s a fixed, non-learned Markov chain — each step depends only on the previous one, and the amount of noise added at each step follows a predetermined schedule, not a learned one.

# Forward process: add noise according to a schedule, one step at a time
def forward_diffusion_step(x, t, betas):
    beta_t = betas[t]
    noise = torch.randn_like(x)
    x_noised = torch.sqrt(1 - beta_t) * x + torch.sqrt(beta_t) * noise
    return x_noised, noise  # keep the noise — it's the training target

Because each step is just Gaussian addition, there’s a closed-form shortcut: you can jump directly from the clean image to any noise level t in a single computation, without looping through every intermediate step. This is what makes training tractable — you sample a random timestep and a random image, noise it once, and that’s one training example.

The Reverse Process: Learning to Denoise

The reverse process is where the actual learning happens. At each noised timestep, the model is given the noisy image and the timestep, and it’s trained to predict the noise that was added, so it can be subtracted back out.

# Reverse process, one step: model predicts noise, we remove a fraction of it
def reverse_diffusion_step(model, x_t, t, betas, alphas_cumprod):
    predicted_noise = model(x_t, t)
    alpha_t = alphas_cumprod[t]
    x_prev = (x_t - torch.sqrt(1 - alpha_t) * predicted_noise) / torch.sqrt(alpha_t)
    return x_prev

Generation, then, is just running this in reverse from pure noise: start at t = T with random Gaussian noise, predict and remove a bit of it, land at t = T-1, repeat down to t = 0. Each step removes a little more uncertainty, and by the final step what remains looks like a photograph, a painting, or whatever the training distribution taught the model to produce.

Noise Schedules

The schedule of how much noise gets added at each forward step — the sequence of beta values above — has a real effect on output quality. Early diffusion work used a linear schedule, adding noise at a constant rate across all timesteps. Later work found that a cosine schedule, which adds noise more slowly near the start and end and faster through the middle, preserves useful signal for longer and produces measurably better samples for the same number of steps. This is a small, easy-to-miss detail in the training recipe that turned out to matter more than it looked like it should.

U-Net vs. Diffusion Transformer

The original architecture for the noise-prediction model was a U-Net: a convolutional encoder-decoder with skip connections between matching resolutions, borrowed almost directly from image segmentation work. It processes the noisy image at multiple resolutions, downsampling then upsampling, and the skip connections let fine detail from early layers reach the later layers that reconstruct it.

More recent large-scale image and video models increasingly replace the U-Net with a Diffusion Transformer (DiT): the image is chopped into patches, treated as a sequence like tokens in a language model, and processed with standard transformer self-attention blocks instead of convolutions. The appeal is the same reason transformers won everywhere else — they scale predictably with more data and more parameters, and the same attention mechanism that lets a language model relate distant words lets a DiT relate distant regions of an image, which matters a great deal once you extend the idea to video.

Property	U-Net	Diffusion Transformer (DiT)
Core operation	Convolution, multi-resolution	Self-attention over image patches
Scaling behavior	Diminishing returns past a point	Scales more predictably with size/data
Inductive bias	Strong (locality, translation invariance)	Weaker, learned from data instead
Common use	Earlier-generation image diffusion	Recent large-scale image and video models

Latent Diffusion: Working in Compressed Space

Running the full noise/denoise process directly on raw pixels is expensive — a high-resolution image has millions of pixel values, and every one of the many denoising steps has to process all of them. Latent diffusion sidesteps this by first compressing the image with a separately trained autoencoder (a VAE) into a much smaller latent representation, running the entire diffusion process in that compressed space, and only decoding back to pixels at the very end.

image (512×512×3)
   → VAE encoder → latent (64×64×4)
   → [diffusion forward/reverse process happens here, entirely in latent space]
   → VAE decoder → image (512×512×3)

This is the architectural choice that made high-resolution image generation practical on consumer hardware: the diffusion model itself only ever has to model a latent grid a fraction of the size of the final image, and the VAE handles the (comparatively cheap, one-shot) job of expanding that back to pixels.

Classifier-Free Guidance: Steering Generation with Text

None of the above explains how a text prompt actually influences the output. The mechanism is classifier-free guidance (CFG): during training, the model occasionally gets the text conditioning and occasionally gets a null/empty condition instead, so it learns to predict noise both with and without knowing what you asked for.

At generation time, the model runs both predictions at every step — conditioned on your prompt, and unconditioned — and extrapolates past the conditioned prediction, away from the unconditioned one:

def cfg_step(model, x_t, t, prompt_embedding, guidance_scale=7.5):
    noise_uncond = model(x_t, t, condition=None)
    noise_cond = model(x_t, t, condition=prompt_embedding)
    # push further in the direction the prompt moves the prediction
    return noise_uncond + guidance_scale * (noise_cond - noise_uncond)

A guidance_scale of 1.0 would just use the conditioned prediction as-is; pushing it higher (typically somewhere in the 5-15 range) exaggerates the prompt’s influence, trading some diversity and occasionally some realism for stronger prompt adherence. This is the parameter behind the “CFG scale” slider you’ll see exposed in most image-generation tools, and it’s a direct, mechanical consequence of how the model was trained with dropped-out conditioning, not a separate bolt-on feature.

From Images to Video: Extending Diffusion Across Time

Video is, structurally, a stack of images with a strong dependency between adjacent frames — and the most direct way to extend diffusion to video is to add a temporal dimension to the same architecture. Instead of denoising a single 2D latent, the model denoises a 3D volume (height, width, time), and the network gains temporal attention layers alongside the existing spatial ones, so a patch in frame 40 can attend to the corresponding region in frame 39 and frame 41, not just other patches within its own frame.

This is what keeps a generated video from flickering into a different-looking scene every few frames: the temporal attention gives the model a mechanism to enforce consistency of objects, lighting, and identity across time, the same way spatial self-attention enforces consistency across a single image. The computational cost scales up sharply, though — you’re now denoising many frames’ worth of latents jointly rather than one image, which is a large part of why video generation remains far more compute-intensive than still images, even with all the same latent-space and architectural tricks applied.

Key Takeaways

• Diffusion models are trained by learning to reverse a fixed, non-learned noising process, not by learning to draw directly.
• The forward process adds Gaussian noise on a schedule; the reverse process is a learned model predicting and removing that noise step by step.
• Cosine noise schedules generally outperform linear ones by preserving signal longer through the middle of the process.
• U-Nets were the original noise-prediction architecture; Diffusion Transformers (DiT) increasingly replace them in large-scale models for their more predictable scaling.
• Latent diffusion runs the entire process in a VAE-compressed latent space, which is what makes high-resolution generation computationally practical.
• Classifier-free guidance conditions the model on your prompt during training some of the time and on nothing the rest of the time, then extrapolates between the two at generation time to control prompt adherence.
• Video diffusion adds temporal attention across frames to the same core architecture, which is what keeps generated video visually consistent over time, at a steep compute cost.