RAG, explained from scratch

A language model only knows what it saw during training — not your school's handbook, not last week's news, not your private notes. Ask it about those and it may confidently make something up (a hallucination). Retrieval-Augmented Generation (RAG) fixes this: it looks up the relevant facts first, then asks the model to answer using them. It's how most real-world "chat with your documents" apps actually work. Let's build the pipeline.

The three steps: Retrieve → Augment → Generate

The name says it all:

1. Retrieve — find the chunks of your documents most relevant to the question.
2. Augment — paste those chunks into the prompt alongside the question.
3. Generate — the LLM answers, grounded in the text you handed it.

The model isn't recalling from memory; it's reading the notes you just gave it and summarising. That's the whole idea.

Setup: turn documents into searchable meaning

Before any question is asked, you prepare your knowledge base:

1. Chunk — split documents into bite-sized passages (a few sentences each).
2. Embed — turn each chunk into a vector using an embedding model, so its meaning becomes a point in space.
3. Store — keep all those vectors in a vector database built for fast "find the nearest points" search.

Query time: find the closest meaning

When a user asks a question:

1. Embed the question into a vector the same way.
2. Find the nearest chunks by cosine similarity — the same closeness measure from the embeddings post:

$\cos\theta = \frac{a \cdot b}{\lVert a\rVert\,\lVert b\rVert}$

The top-$k$ chunks (say, the 3 highest-scoring) are the ones whose meaning best matches the question — even if they don't share the exact keywords.

3. Augment: build a prompt like "Using the passages below, answer the question. Passages: [chunk 1] [chunk 2] [chunk 3]. Question: …"
4. Generate: the LLM produces an answer grounded in those passages — and can even cite them.

A tiny worked intuition

Imagine three stored chunks with embedding vectors, and a question embedding $q$. You score each by cosine similarity:

chunk A (refund policy)   cos(q, A) = 0.83   ← top match
chunk B (opening hours)   cos(q, B) = 0.41
chunk C (staff bios)      cos(q, C) = 0.12

The question ("Can I get my money back?") lands closest to the refund chunk, so that text gets pasted into the prompt — and the model answers from real policy instead of guessing.

Why RAG matters

• Fewer hallucinations — the model answers from provided facts, not fuzzy memory, and can quote sources.
• Fresh & private knowledge — update the documents, not the model; no expensive retraining to add today's data or your internal wiki.
• Cheaper than fine-tuning — you change what the model reads, not what it is.

It's not magic, and it has real engineering depth (how you chunk, whether you add keyword search alongside vectors, how you rerank the results, how you evaluate faithfulness). But the backbone is always the same three steps: retrieve the right text, put it in the prompt, let the model answer from it.

Now watch the pipeline. The demo below walks a query through retrieve → augment → generate, so you can see each stage light up.

RAG is built on embeddings and tokenization — both explained from scratch here. Ready to build one? Try a free lesson.