The chain rule and backpropagation

Training a neural network means answering one question, over and over: "If I nudge this weight a little, does the error go up or down, and by how much?" Answer it for every weight and you know exactly how to improve the whole network. The algorithm that answers it — efficiently, for millions of weights at once — is backpropagation. And backprop is really just one idea from calculus: the chain rule. Let's build it.

Networks are nested functions

A neural network is functions stuffed inside functions: the input feeds layer 1, whose output feeds layer 2, and so on, until a final loss measures how wrong the answer was. To improve, we need the gradient — the slope of the loss with respect to each weight (which way, and how steeply, the error changes as we wiggle that weight).

The problem: a weight in an early layer affects the loss only through all the layers after it. How do you get its slope? The chain rule.

The chain rule

If a quantity $z$ depends on $y$, and $y$ depends on $x$, then the slope of $z$ with respect to $x$ is the product of the slopes along the way:

$\frac{dz}{dx} = \frac{dz}{dy}\cdot\frac{dy}{dx}$

In words: to see how $x$ affects $z$, multiply "how $x$ affects $y$" by "how $y$ affects $z$." Rates of change multiply through a chain.

Worked example

Let $y = 3x$ and $z = y^2$. So $z$ depends on $x$ through $y$.

By the chain rule:

dz/dy = 2y          (slope of y²)
dy/dx = 3           (slope of 3x)
dz/dx = 2y × 3 = 6y = 6(3x) = 18x

Check directly: substitute to get $z = (3x)^2 = 9x^2$, whose slope is $dz/dx = 18x$. ✓ Same answer.

At $x = 2$: the chain gives $dz/dx = 18\times2 = 36$. So nudging $x$ up by a tiny 0.01 raises $z$ by about $0.36$. That single number — the gradient — is exactly what training needs.

Backpropagation = the chain rule, run backward

A real network has thousands of these links. Backprop computes all the gradients in two passes:

1. Forward pass. Feed the input through the layers, computing and remembering each intermediate value, until you get the output and the loss.
2. Backward pass. Start at the loss and walk backward, multiplying local slopes (chain rule) layer by layer. Each weight receives its gradient — its share of the "blame" for the error.

The magic is reuse: the gradient flowing into layer 3 is used to compute layer 2's gradients, which are used for layer 1's. Nothing is computed twice. That efficiency is what makes training billion-parameter models possible at all.

Those gradients then feed gradient descent, which nudges every weight a step downhill. Backprop finds the direction; gradient descent takes the step. Together they're the whole training loop.

Why this matters

• Every deep-learning framework (PyTorch, TensorFlow) is, at its core, an autograd engine that runs backprop for you — you just write the forward pass.
• "Vanishing gradients" (why very deep networks once failed to train) is just the chain rule multiplying many small slopes into near-zero — understanding backprop explains the fix (ReLU, residual connections, normalization).
• It's the reason a network can learn from its mistakes at all.

Now watch it flow. In the demo below, values move left-to-right in the forward pass; trigger the backward pass and follow the gradients flowing right-to-left — each node's gradient built from the ones after it. That's the chain rule, live.

---

Want to backprop a tiny network by hand and feel it click? Try a free lesson.