Anatomy · Language models · Working model

GPT-2, running in your browser

This is GPT-2: the same tokenizer algorithm, the same block, the same wiring, scaled down from 124 million parameters to 940,800 so it fits in a page, and trained on children's stories so that something that small can still write. Type anything. Every number below is computed in your browser by the model, and the machine moves because of those numbers.

Loading the weights (966 KB)…
436·upon258·a395·time44,400·there282·was258·a388·little·girl·boy·dog·bird·m
The token being computed, its pegs, and the drawn next tokenFour tiers: the four transformer blocksThreads: attention, darker is strongerEach tier's threads run from earlier tokens to the selected one, weighted by that block's attention averaged over its heads (pick one head in panel 3). Its dial reads the MLP's write, 0 to 2 times the stream's length. The plungers rise to the top five next-token probabilities on a 0 to 1 scale; tokens that slide off the left of the rail drop into the galley but stay in the context.
Tokenize

Text becomes a row of integers

A regex first splits the text into words, numbers and punctuation, each carrying its leading space. Each piece is spelled out as raw bytes, then merged pair by pair in the order the merges were learned: the earliest-learned pair that is present always merges first. What is left is a list of vocabulary entries, and their row numbers are the tokens.

GPT-2 learned 50,000 merges on 40 GB of web text; this one learned 767 on TinyStories, so common story words are a single token and anything else falls back to pieces. Because it starts from bytes, no input is ever out of vocabulary.

Go deeper Byte-level BPE Tokenisation and BPE Training a tokeniser Special tokens
Embed

Each token looks up a row, and so does its position

The token id picks one row of a 1,024 × 128 table; the position picks one row of a 128 × 128 table. Their sum is the token's starting state, a vector of 128 numbers that every later stage reads and adds to: the residual stream.

Nothing told the model which words are alike. Training pushed rows of words used in similar places to point in similar directions, which is why the nearest rows below tend to make sense.

Go deeper Embedding lookup Positional encodings The residual stream Static vs contextual
Attend

Each token gathers from the ones before it

In every block, each position makes a query, a key and a value. A position's query is compared with the keys of every position up to and including itself, the scores are scaled by √dhead and pushed through a softmax, and the weights mix the values. The mask means a token can never see the future, which is what makes the model a predictor.

Each block runs 4 heads in parallel on 32-wide slices, so each can learn a different habit: look at the previous token, look back at a name, look at the start. Pick a block and a head; the threads on the machine follow your choice.

Go deeper Attention Soft dictionary lookup Multi-head attention Causal masking Attention sinks
Transform

An MLP rewrites each position on its own

After attention, each position passes through a two-layer network on its own: 128 up to 512 numbers, a GELU, and back down to 128. The result is added to the stream, never written over it. Two thirds of every block's parameters live here, and most of what a model knows about words is thought to be stored in these weights.

The dial on each tier of the machine is this stage's write: how far the MLP moves the selected token's vector, as a share of the vector's length.

Go deeper The transformer block LayerNorm and residuals Residual connections
Read out

Scores for every token in the vocabulary

After the last block, a final LayerNorm and one matrix multiply score all 1,024 tokens. GPT-2 reuses the embedding table for this (weight tying): the score of a token is how well the final vector lines up with that token's own row.

Apply the same read-out to the stream after each earlier block and you can watch the guess form. This is the "logit lens": early blocks often guess something generic, and later blocks sharpen it.

Go deeper Unembedding and the logit lens Weight tying Softmax and logits Next-token prediction
Choose

Pick one, append it, run again

The scores become probabilities through a softmax. Temperature divides the scores first: below 1 sharpens the distribution, above 1 flattens it, and 0 always takes the top token. Top-k then keeps only the k most likely and renormalises. A seeded random draw picks one token; it is appended to the text, and the whole pass runs again for the next one. That loop is all "generation" is.

The five plungers on the machine are the five most likely next tokens at your settings; the lit one is the draw.

Go deeper Temperature Top-k sampling Nucleus sampling Greedy decoding Autoregressive generation
Theory

The whole forward pass, in seven lines

For a sequence of token ids t1…tn, with WE the token table and WP the position table:

x⁰ᵢ = W_E[tᵢ] + W_P[i] embed and add position for each block ℓ = 1 … 4: a = Attn(LN₁(xℓ⁻¹)) each head: softmax(QKᵀ/√d_h + mask)·V, heads concatenated, projected h = xℓ⁻¹ + a residual add xℓ = h + W₂ · GELU(W₁ · LN₂(h) + b₁) + b₂ MLP, 128 → 512 → 128, residual add logits = LN_f(xᴸₙ) · W_Eᵀ tied read-out, one score per token p = softmax(logits / T), keep top-k, draw tₙ₊₁ then append and repeat

That is all of it. GPT-2 small, medium, large and XL differ from each other, and from this miniature, only in the four sizes: blocks, width, heads and context. The table recomputes each model's parameter count from those sizes with the same formula, and the first row is checked against the tensors this page actually loaded.

Theory

Where the parameters are

Per block: the attention projections hold 4·d² (query, key, value and output) and the MLP holds 8·d² (two d × 4d matrices), plus small biases and LayerNorm gains. So a block is about 12·d² parameters, two thirds of them in the MLP.

At d = 128 the embedding table is a large share of this model (16%); in GPT-2 small it is 31%, and in GPT-2 XL under 6%, because blocks grow with d² and depth while the table grows only with d. A model this small spends a third of its weights just naming tokens.

Counting transformer parameters · FLOPs of a forward pass
Honesty

What this miniature can and cannot do

It has read about 80 million tokens of simple stories, nothing else. Its validation loss is 1.73 nats per token (perplexity 5.6), so it reliably continues a story with names, plain grammar and a small vocabulary, and it has no facts about the world: ask it for a capital city and it will tell you about a park.

Its weights are stored as 8-bit integers with one scale per row, which cost 0.001 nats of loss against the float32 original, measured on the same validation batches. The architecture and every computation are GPT-2's; only the sizes and the data differ.

Perplexity · Small language models · KV cache
How it was made

Trained for this page, in 52 minutes on a laptop CPU

The tokenizer is byte-level BPE trained on the TinyStories validation split with GPT-2's pre-tokenisation pattern and byte table. The model is GPT-2's architecture as written in nanoGPT (pre-LayerNorm, tanh-GELU, learned positions, tied read-out, GPT-2's scaled initialisation of the residual projections), trained with AdamW for 9,754 steps of 64 × 128 tokens on the first 120 MB of the TinyStories GPT-4 training split. The page reimplements the forward pass in plain JavaScript; its logits are checked against PyTorch's for fixed prompts before the page is built.

Sources

  1. Radford et al., 2019. Language Models are Unsupervised Multitask Learners (GPT-2)
  2. Vaswani et al., 2017. Attention Is All You Need
  3. Eldan and Li, 2023. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (data, CDLA-Sharing-1.0)
  4. Sennrich, Haddow and Birch, 2016. Neural Machine Translation of Rare Words with Subword Units (BPE)
  5. Hendrycks and Gimpel, 2016. Gaussian Error Linear Units
  6. nostalgebraist, 2020. Interpreting GPT: the logit lens
  7. Karpathy, nanoGPT (the reference implementation the training script follows)

Built with the anatomy kit · every number on this page is computed from the loaded weights