How an LLM predicts the next word

Ask ChatGPT to write a story and it feels like it planned the whole thing. It didn't. Under the hood, a large language model (LLM) does one small thing, over and over: guess the next chunk of text. That's the entire trick. Let's see exactly how — with real numbers you can check by hand.

Text is made of "tokens," not words

A model doesn't work with letters or whole words. It works with tokens — chunks of text that are usually a word or a piece of a word. "cat" might be one token; "unbelievable" might split into "un", "believ", "able". The model has a fixed list of every token it knows, called its vocabulary (often 50,000+ tokens).

The loop: score → probabilities → pick → repeat

Given everything so far ("The cat sat on the ___"), the model produces one score for every token in its vocabulary. A high score means "this token fits well here." These raw scores are called logits.

But scores aren't probabilities. To turn them into "there's a 69% chance the next word is mat," we use a function called softmax, which does two things:

1. makes every number positive (using the exponential eˣ, where e ≈ 2.718), and
2. divides each by the total, so they all add up to 100%.

In symbols, for a list of scores $s_1, s_2, \dots$, the softmax of the $i$-th score is:

$\text{softmax}(s)_i = \frac{e^{s_i}}{\sum_{j} e^{s_j}}$

Worked example

Say only three tokens have meaningful scores after "The cat sat on the ___":

mat  → score 3.0
sofa → score 2.0
moon → score 0.5

Softmax, step by step:

e^3.0 ≈ 20.09
e^2.0 ≈ 7.39
e^0.5 ≈ 1.65
total ≈ 29.13

P(mat)  = 20.09 / 29.13 ≈ 0.69   (69%)
P(sofa) =  7.39 / 29.13 ≈ 0.25   (25%)
P(moon) =  1.65 / 29.13 ≈ 0.06   (6%)

The model now samples one token using these odds — usually "mat," sometimes "sofa," rarely "moon." Then it adds that token to the sentence and runs the whole thing again to pick the word after it. One token at a time, forever. There is no master plan — just a very, very good next-token guess, repeated.

Temperature: the creativity dial

Before softmax, we can divide every score by a number T called the temperature. It controls how bold the model is.

Low temperature (T = 0.5) — divide scores by 0.5, i.e. double them → [6, 4, 1]:

P(mat) ≈ 88%,  P(sofa) ≈ 12%,  P(moon) ≈ 0.6%

The model gets sharper — it almost always picks the top choice. Safe and repetitive.

High temperature (T = 2) — halve the scores → [1.5, 1, 0.25]:

P(mat) ≈ 53%,  P(sofa) ≈ 32%,  P(moon) ≈ 15%

The odds flatten — riskier, more surprising words get a real chance. Creative, but more likely to wander.

That's the whole reason the same prompt can give different answers, and why "make it more creative" is literally a slider.

Why this matters

Everything an LLM does — answering, translating, coding, writing — is this next-token loop wearing different hats. Understanding it demystifies a lot: hallucinations are just confident-but-wrong guesses; longer replies cost more because each token is a full pass; "temperature" is a real knob, not a metaphor.

Now play with it. Watch the probability bars shift as you drag the temperature — then predict which token wins before you let go.

---

Want to build these ideas by hand instead of just reading? Try a free lesson — you'll compute a softmax yourself and see it click.