Gradient descent, by hand

Every machine-learning model — from a simple line-fit to ChatGPT — learns the same way: it starts out wrong, measures how wrong, and nudges itself a little less wrong, over and over. That nudging algorithm is called gradient descent, and it's simpler than it sounds. Let's do it by hand.

Learning = minimizing an error

When a model makes predictions, we score how bad they are with a loss (also called error or cost) — one number, where smaller is better. Training is just: find the settings that make the loss as small as possible.

Picture the loss as a valley. The horizontal position is a setting you can change; the height is how wrong you are. You want to reach the bottom. But you can't see the whole valley — you can only feel the slope under your feet. So the plan is obvious:

Feel which way is downhill, take a small step that way, repeat until the ground is flat.

The "slope under your feet" is the gradient. Stepping opposite the gradient (downhill) is gradient descent.

The update rule

If x is your current setting and the slope there is gradient, the next setting is:

$x_{\text{new}} = x_{\text{old}} - (\text{learning rate}) \times \text{gradient}$

The learning rate is how big a step you take. That's the entire algorithm.

Worked example

Let's minimize the simplest valley: $\text{loss} = x^2$. Its slope at any point is $2x$ (that's the derivative — for $x^2$, the slope is $2x$; you can trust it for now, or check that the curve is steeper the farther you are from 0).

Start at x = 3 with a learning rate of 0.1:

Step 1:  gradient = 2 × 3   = 6      → new x = 3    − 0.1×6   = 2.40
Step 2:  gradient = 2 × 2.4 = 4.8    → new x = 2.40 − 0.1×4.8 = 1.92
Step 3:  gradient = 2 × 1.92= 3.84   → new x = 1.92 − 0.1×3.84= 1.536

Watch the loss shrink as we go:

x = 3.00  → loss = 9.00
x = 2.40  → loss = 5.76
x = 1.92  → loss = 3.69
x = 1.536 → loss = 2.36

We're sliding toward x = 0, the bottom of the valley, exactly as we should. Keep stepping and it gets closer and closer.

The learning rate is everything

Take too big a step and you overshoot the bottom and fly up the other side. Same valley, but learning rate 1.1:

Step 1:  gradient = 2 × 3 = 6  → new x = 3 − 1.1×6 = −3.6

We started 3 away from the bottom and landed 3.6 away — farther than before. Loss went from 9 to 12.96. Keep going and it explodes. This is divergence, and it's the #1 reason training fails. Too small a learning rate is safer but painfully slow. Getting it right is the single most important dial in training.

Why this matters

Real models don't have one setting x — they have millions or billions (the weights). But the idea is identical: compute the slope of the loss with respect to every weight (that giant slope-vector is the gradient), then step them all a little downhill. The clever algorithm that computes all those slopes efficiently is backpropagation — but the step it feeds is exactly the rule you just used by hand.

Now feel it. Press Step below and watch the point roll into the valley. Then crank the learning rate up and watch it overshoot and bounce — the divergence we just computed, live.

---

Reading about it isn't the same as doing it. Try a free lesson and run gradient descent yourself, one step at a time.