GPT,
made tangible.
Build it. Measure it. Train it.
One guide. Every operation.
Turn the tensors in real 3D, follow the arithmetic, and train a tiny GPT at your own pace.
Start Chapter 5 → · Build the model first →
Drag to orbit · scroll or pinch to zoom · click a cell to inspect
text → [6]Open the PyTorch code for this step
Code builds on the preceding operations and the book’s classes. The browser uses the same arithmetic in JavaScript. Tables round values for readability; select a cell for its full stored precision.
Inspect all 568 fixed tiny-GPT parameters
Explicit illustrative values, never trained. Linear weights use [output, input] order. These are the exact parameters used by the bridge and Chapter 4 model.
Run the complete tiny GPT in PyTorch
Copy this complete script into a Python file with PyTorch installed. It loads the exact browser weights, prints the parameter count, and reproduces all eight generated tokens. Dropout is disabled by model.eval().
Run Chapter 5: train, evaluate, save and sample in PyTorch
A complete standalone script using the same starting weights and two training examples. It updates all 568 parameters with AdamW, prints train/validation losses for 25 updates, writes tiny-gpt-checkpoint.pth in your working directory, reloads it and samples tokens. Dropout is set to zero for reproducibility. Browser snapshots stay in tab memory; run this script locally to save a file.
A small model of a big idea.
Two heads, two blocks, four hidden coordinates and eight vocabulary entries. Chapters 2–4 explain the starting model; Chapter 5 measures its loss and trains all 568 parameters with real gradients. All calculations run in your browser. The held-out examples and pretrained-weight mapping are teaching demonstrations, not a language benchmark or a loaded GPT-2. Nothing plays automatically.
Where do the numbers come from?
The original eleven lessons retain the six input vectors from Sebastian Raschka’s Chapter 3 example and their illustrative projections. The bridge introduces a separate width-four example so two heads divide evenly. Chapter 4 uses the same tiny vocabulary and fixed, fully inspectable weights. Its operations follow the author’s GPT implementation and are checked against independent PyTorch reference calculations.
How to use the 3D view
Drag to orbit actual 3D geometry; scroll or pinch to zoom. Camera buttons provide the same controls from a keyboard. Click a cell to inspect its full value. Labels that would overlap are hidden; use Flat for every value and keyboard cell selection. Negative values extend below the plane; forbidden scores show −∞ symbolically. ID blocks have equal heights because IDs are addresses, not quantities.
What happened to the separate walkthrough?
Its foundations and worked attention examples remain here, with their original lesson links. Choose a chapter, open one operation, and move forward when you are ready.