How diffusion models paint from noise

Type "an astronaut riding a horse" into Stable Diffusion, DALL·E, or Midjourney and a brand-new image appears. Where does it come from? The surprising answer: the model starts from pure random static and slowly cleans it up into a picture. That's diffusion. Let's see exactly how — including the maths.

Two directions: destroy, then learn to rebuild

Diffusion has a forward process (easy, no learning) and a reverse process (the part the model learns).

Forward (add noise). Take a clean training image and add a tiny bit of random noise. Repeat, say, 1000 times. Early on it still looks like the photo; by the end it's indistinguishable from TV static. This is trivial — anyone can add noise.

Reverse (remove noise). Now train a neural network to undo one step of that: given a slightly-noisy image, predict the noise that was added, and subtract it. Do that well enough and you can start from pure static and denoise, step by step, until a clean image emerges. That reverse walk is how new images are generated.

The clever insight: learning to destroy is easy, so use it to teach the model to create. The forward process hands you unlimited "noisy → clean" training pairs for free.

The forward step, in maths

One forward step mixes a little noise into the image. If $x_{t-1}$ is the image at step $t-1$, the next, slightly-noisier image is:

$x_t = \sqrt{1-\beta}\;x_{t-1} + \sqrt{\beta}\;\varepsilon$

• $\varepsilon$ ("epsilon") is fresh random noise (drawn from a bell curve).
• $\beta$ ("beta") is a small number — the amount of noise added this step (e.g. 0.02).
• $\sqrt{1-\beta}$ keeps most of the previous image; $\sqrt{\beta}$ sprinkles in a little noise.

Because $\beta$ is small, each step barely changes the image — but stacked over 1000 steps, the original signal fades to almost nothing and randomness takes over.

A tiny worked intuition

Think of one pixel as a mix of signal (the true picture) and noise. As steps pile up, the signal's share shrinks and noise's share grows:

step 0    → 100% picture,   0% noise   (the clean image)
step 250  →  75% picture,  25% noise   (grainy but recognisable)
step 500  →  50% picture,  50% noise   (half-dissolved)
step 1000 →  ~0% picture, ~100% noise  (pure static)

Generation runs this backwards: start at "100% noise," and at each step the network nudges the mix back toward "more picture," until a clean image appears. The $\sqrt{\ }$ in the formula is just the exact bookkeeping that keeps the total "amount of stuff" steady as you trade signal for noise.

Why diffusion won

Before diffusion, the best image generators were GANs. Diffusion overtook them because:

• It's stable to train. GANs pit two networks against each other and often collapse; diffusion just learns "predict the noise," a calm, ordinary training task.
• It's diverse. GANs tend to produce a narrow set of images (mode collapse); diffusion samples freshly from noise each time, so it explores far more of the space.
• It's controllable. You can steer the denoising with a text prompt (via attention) or an outline (ControlNet), which is how "astronaut riding a horse" becomes that exact scene.

The same idea now powers video and audio generation, too — always the same recipe: learn to remove noise, then start from noise and remove it.

Now scrub it yourself. Drag the slider below to dissolve a clean image into static and back — the forward and reverse processes, in your hands.

---

Curious how the "predict the noise" network actually learns? That's gradient descent. Ready to build these ideas? Try a free lesson.