Tokenization: how LLMs read text

Before a single word of your prompt reaches ChatGPT, it gets chopped into pieces called tokens and turned into numbers. This step — tokenization — sounds boring, but it quietly explains a lot of confusing model behaviour: why AI is billed per token, why it has a "context limit," and why it sometimes can't spell or count letters. Let's see how it works, by hand.

Tokens: not letters, not quite words

A token is a chunk of text — often a whole word, sometimes a piece of one, sometimes just a space or punctuation mark. The model keeps a fixed list of every token it knows (its vocabulary, often 50,000–100,000 entries) and maps each to a number. Roughly:

• common words → one token each ("the", "cat")
• rarer words → several tokens ("tokenization" might split into "token" + "ization")
• a space usually rides along with the word after it

So "I love pizza" might be 3 tokens, while "antidisestablishmentarianism" could be 6+. The model only ever sees the numbers.

How the vocabulary is built: byte-pair encoding (BPE)

Where does the token list come from? The most common method is byte-pair encoding (BPE), and it's just repeated counting-and-merging:

1. Start with the smallest pieces — individual characters.
2. Count every adjacent pair of pieces across a big pile of text.
3. Merge the most frequent pair into a single new token.
4. Repeat thousands of times.

Frequent combinations bubble up into their own tokens; rare ones stay split. That's why common words become single tokens — they got merged early.

Worked example

Tiny corpus (spaces shown as gaps between words):

low  low  low  lower

Split into characters:

l o w   l o w   l o w   l o w e r

Count adjacent pairs. The pair l o appears 4 times (once per "low"/"lower") — the most frequent. Merge it into a new token lo:

lo w   lo w   lo w   lo w e r

Now the pair lo w appears 4 times — merge into low:

low   low   low   low e r

After just two merges, low is a single token. The common chunk earned its place in the vocabulary; the rare ending e r stayed as small pieces. Run this thousands of times on real text and you get a vocabulary where everyday words are one token and unusual ones break into familiar sub-pieces.

Why this matters (a lot, in practice)

• You're billed per token. A model's work — and API pricing — scales with tokens read and written, not characters. Longer prompts and replies literally cost more.
• Context limits are in tokens. "128k context" means 128,000 tokens, not words — so a long document might use far more of your budget than its word count suggests.
• Spelling & counting quirks. Because the model sees "strawberry" as a couple of tokens, not the letters s-t-r-a-w-b-e-r-r-y, questions like "how many R's are in strawberry?" are genuinely hard for it — it never saw the individual letters.
• Numbers and code tokenize awkwardly, which is part of why models sometimes fumble arithmetic and formatting.

Now watch it split. In the demo below, type or read text and see it break into coloured tokens — notice how common words are one token while rare ones shatter into pieces, and count the tokens to see why "length" is measured this way.

---

Tokens are step one; next the model turns each into an embedding. Want to see it end to end? Try a free lesson.