Embeddings: turning words into numbers

A computer can add 2 + 2, but it can't add king + queen. Words aren't numbers. Yet somehow ChatGPT "knows" that king is close to queen and far from banana. The trick that makes this possible is called an embedding — and it's the quiet foundation under search, recommendations, and every RAG system. Let's build the idea from scratch.

A word becomes a list of numbers

An embedding turns a word into a vector — an ordered list of numbers. You could imagine (a wildly simplified) 2-number embedding where slot 1 = "royalty-ness" and slot 2 = "person-ness":

king  → [0.95, 0.90]
queen → [0.93, 0.92]
banana→ [0.02, 0.01]

Now "meaning" has become position in space. Words that mean similar things get vectors that sit close together; unrelated words land far apart. Real models don't use 2 numbers — they use hundreds (often 768 or more) — but the idea is identical: a point in space per word.

Nobody hand-picks these numbers. The model learns them by reading enormous amounts of text, nudging the vectors until words that appear in similar contexts end up near each other. (As the linguists say: "you shall know a word by the company it keeps.")

Measuring "closeness": cosine similarity

If meaning is position, then similar meaning = pointing in the same direction. The standard way to measure that is cosine similarity — the cosine of the angle between two vectors. It comes straight from the dot product:

$\cos\theta = \frac{a \cdot b}{\lVert a\rVert\,\lVert b\rVert}$

• $a \cdot b$ is the dot product (multiply matching slots, add them up).
• $\lVert a\rVert$ is the length of the vector, $\lVert a\rVert = \sqrt{a_1^2 + a_2^2 + \cdots}$.

Dividing by the lengths cancels out how long the vectors are, leaving only which way they point. The result runs from +1 (same direction — very similar) through 0 (at right angles — unrelated) to −1 (opposite).

Worked example

Take a = [3, 4] and b = [4, 3]:

a · b       = (3×4) + (4×3) = 24
‖a‖         = √(3² + 4²) = √25 = 5
‖b‖         = √(4² + 3²) = √25 = 5
cos(a, b)   = 24 / (5 × 5) = 24 / 25 = 0.96

A cosine of 0.96 (nearly 1) means these two vectors point almost the same way — "very similar meaning." Now compare a = [3, 4] with c = [−4, 3]:

a · c     = (3×−4) + (4×3) = −12 + 12 = 0
cos(a, c) = 0 / (5 × 5) = 0   →  unrelated

A cosine of 0 means they're at right angles — no relationship. This single number is exactly how a search engine decides which documents match your query.

Meaning has directions, too

Here's the famous party trick that made embeddings famous. Because words are vectors, you can do arithmetic on meaning:

$\text{king} - \text{man} + \text{woman} \approx \text{queen}$

Subtracting "man" and adding "woman" moves you along the "gender direction," and you land near queen. The model was never told this — it fell out of arranging words by the company they keep. Relationships (gender, tense, capital-of-country) become consistent directions in the space.

Why this matters

Embeddings are how machines turn messy human language into something they can compute with:

• Semantic search — your query and every document become vectors; the closest ones (by cosine similarity) are the best matches, by meaning, not exact keywords.
• RAG (retrieval-augmented generation) — when an AI "looks up relevant notes" before answering, it's comparing embedding vectors.
• Recommendations — songs, products, and videos get embeddings too; "similar to this" is just "nearby vector."

Now play with it. In the demo below, click a word to make it the query and watch the others re-rank by meaning — the cosine similarity we just computed, live.

---

Reading about it isn't the same as building it. Try a free lesson and compute a similarity yourself — it clicks fast.