What is a Transformer block?

ChatGPT sounds miraculous, but its architecture is almost boringly repetitive: it's one block, stacked dozens of times. If you understand that single block, you understand the shape of every modern large language model. This post assembles it end to end. (For the full derivation of the attention maths, read Attention Is All You Need, explained first — here we snap the pieces together.)

What goes in, what comes out

A block takes a sequence of vectors — one per token, each carrying that token's meaning-so-far — and returns a sequence of the same shape, but with each vector enriched by context from the others. Stack many blocks and the vectors get progressively more "aware" of the whole sentence.

One block does four things, in order:

1. Multi-head self-attention   →  mix information between tokens
2. Add & Normalize             →  residual connection + layer norm
3. Feed-forward network         →  let each token "think" on its own
4. Add & Normalize             →  residual + layer norm again

Let's take them one at a time.

1. Multi-head self-attention — mixing information between tokens

This is the heart. Each token forms a query ("what am I looking for?"), compares it against every token's key (a dot product → a match score), softmaxes those scores into attention weights, and builds a blend of the tokens' values. The result: "it" pulls in the meaning of "the cat," "bank" figures out river-vs-money from its neighbours. The compact formula:

$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V$

Multi-head just runs this several times in parallel with different learned projections, so different "heads" capture different relationships (grammar, meaning, reference) at once — then the heads are combined. This is the only step where tokens talk to each other.

2 & 4. Add & Normalize — the safety wires

After attention, we don't replace the input — we add it back:

$\text{output} = x + \text{Attention}(x)$

This is a residual connection. It means information is never lost, and the signal can flow cleanly through dozens of stacked blocks without fading (the vanishing-gradient problem from the backprop post). Then layer normalization rescales the numbers to a steady range so the next step gets a clean, well-behaved signal. Think: "keep a photocopy and add it back, then adjust the volume."

3. Feed-forward network — per-token thinking

Attention shared information across tokens; now each token processes what it gathered, on its own, through a tiny two-layer network:

$\text{FFN}(x) = \max\!\left(0,\; x W_1 + b_1\right) W_2 + b_2$

Multiply by a learned matrix, apply ReLU (zero out negatives), multiply again. This is where a lot of the model's learned "knowledge" actually lives.

Stack them — and you have GPT

That's the whole block: attention → add&norm → feed-forward → add&norm. Stack $N$ of them (GPT-style models use dozens to over a hundred), add token embeddings and positional encodings at the bottom, and a "predict the next token" head at the top (how LLMs generate), and you have a Transformer — the architecture behind essentially every modern LLM.

• Encoder-style blocks (like BERT) let every token see every other → great for understanding text.
• Decoder-style blocks (like GPT) mask future tokens so the model can only look backward → great for generating text one token at a time.

Same block, one masking tweak, and you've covered most of modern NLP.

Why this matters

The Transformer's superpower is that it's parallel (all tokens processed at once — fast on GPUs) and scalable (add blocks, heads, data → it keeps improving). That scalability is the entire reason GPT-2 → GPT-3 → GPT-4 kept getting dramatically better. Understand one block and the frontier stops being magic.

Now explore attention — the block's core — in the demo below: click a token and watch which others it draws from.

---

This ties together attention, embeddings, ReLU and backprop — all built from scratch in this blog. Ready to build a block yourself? Try a free lesson.