Tokenization, visualized
Language models don't read letters or whole words — they read tokens: chunks of text that might be a word, part of a word, or a punctuation mark. Before any text reaches the model, a tokenizer splits it into these pieces and maps each to a number.
This matters in practice: pricing, context limits, and even how a model spells are all measured in tokens. Seeing text broken into tokens explains a lot of otherwise-confusing model behaviour.
See text split into tokens
Example text
“transformers are unbelievably powerful!”
Token count
9 tokens
Amber chips (##) are sub-word continuations of the previous piece — one word can cost several tokens.
Models bill and remember in tokens, not words. That's why prompt length, context windows and costs are all measured this way.
Free · runs entirely in your browser · nothing to install
How to use it
- Type or read the sample text and watch it split into coloured tokens.
- Notice how common words are one token but rare words break into several.
- Count the tokens to see why 'length' is measured in tokens, not characters.
What you'll take away
- What a token is and why it isn't the same as a word.
- Why context windows and API costs are counted in tokens.
- How tokenization explains quirks in model outputs.
Want to actually build this?
This demo is one moment inside a full Math to Machine lesson — predict, build, and explain the concept, with an AI tutor that gives hints, not answers. The first five lessons are free.
FAQ
- What is a token in an LLM?
- A token is a small chunk of text — often a word or a piece of a word — that the model treats as a single unit. Text is split into tokens before the model processes it.
- Why do AI models charge per token?
- The model's work scales with the number of tokens it reads and writes, so providers measure usage in tokens. Longer prompts and responses use more tokens and therefore cost more.