Build a small GPT
Take this if you have read that transformers changed everything and want to write one rather than read another diagram of one.
What you’ll be able to do
You can implement self-attention from scratch, explain what queries, keys and values each do, and train a character-level transformer that generates text in the style of a corpus you supplied.
What it assumes
Confident Python, comfort with matrix multiplication, and having trained at least one small neural network before.
- Price
- ₹499
- Length
- 3 weeks
- Sessions
- 12
- Effort
- ~6 h/week · 18 h total
- For
- College · Class 11–12 · Working professional
Adds it to My courses on your dashboard so you can find it again. Payment is not open yet — nothing is charged.
What’s taught, in order
- 01
Attention as a lookup you weight
Computing attention scores by hand on a five-token sentence before any code, so queries, keys and values arrive as roles in a calculation you have performed rather than as three mysterious matrices.
- 02
Self-attention in code
Implementing single-head attention with explicit loops, then vectorising it, then adding multiple heads. Each step is checked against the hand computation from module one.
- 03
A block, and then a stack
Residual connections, layer norm and the feed-forward layer — what each one is preventing, discovered by removing it and watching training fail in its own characteristic way.
- 04
Tokenizing, training, generating
A character-level vocabulary, a training loop you can watch, and sampling with temperature. Ending on generation means you finish with something readable that you made.
What you build
Train your character-level transformer on a corpus you choose — song lyrics, a public-domain book, your own writing, cricket commentary. Ship the generated samples at three training checkpoints, your loss curve, and a short note on what the model learned first, what it learned last, and what it never learned.
Check it yourself against this
- Your attention implementation matches your hand-computed values on the module-one example.
- The loss curve shows genuine training, and you can explain any flat region in it.
- Samples from three checkpoints show a visible progression in quality.
- You can name something structural the model learned early, such as spacing or line length.
- You can name a pattern it never learned, with a reason grounded in the model's size or data.
Then answer this
Your model has no notion of a word, only characters, yet it produces mostly real words — where does that knowledge live?
Two or three sentences, in your own words. If you cannot, the course is not finished — go back to the module it came from.
What this course does not do
This builds a small model on a small corpus: expect text that is locally plausible and globally incoherent. Pretraining at scale, instruction tuning, RLHF and inference optimisation are not here, and the result will not be useful for anything except understanding, which is the entire point.
Where this leads
The attention visualiser is free and standalone — watching which tokens attend to which in a real sentence is the clearest possible preview of the mechanism this sprint asks you to implement.
FAQ
- Is this the same as Karpathy's nanoGPT?
- Same spirit and a similar destination, with more scaffolding on the way. The difference is the hand-computation first module and the checks against it — you verify your own attention implementation against numbers you worked out on paper rather than against a reference repository.
- How much compute do I need?
- A laptop. The model is small and the corpus is a few hundred kilobytes, so training runs take minutes rather than hours. Nothing here needs a GPU or a cloud account.
- Can I take this without having done the CNN sprint?
- Yes, they are independent. What you do need is one prior experience of training a neural network and watching a loss curve, because this sprint assumes you already know what a training loop feels like.
Other courses
This is a standalone course. It does not add weeks to the 78-week curriculum or change your roadmap. If you want the whole path instead, the full programme is ₹399/month · ₹2,000/year — see what it covers.