Dissection · Vaswani et al., 2017

Anatomy of the Transformer

Attention Is All You Need, dissected: the whole encoder-decoder in 3D, every matrix a real one from a trained model running in your browser, every number from the paper traced to its sentence.

PaperVaswani et al., NIPS 2017

Model61,472 parameters, live

Reading24 min · 29 steps

Published2026-10-15

01Overview

A translation machine with no recurrence in it

In June 2017 eight researchers at Google proposed a sequence-to-sequence model "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely" [1]. It translated English into German better than any model before it, ensembles included, and trained in a fraction of the time. Every large language model since is a descendant.

What you are looking at is that model, whole: the encoder on the left, the decoder on the right, the embedding matrix they share at the bottom, the output layer at the top. It is not a diagram. Every block is a real matrix from a trained network running in this page, and every cell is one number in it: its colour and height are the value. Hover any cell to read it.

The network here is the paper's design at one sixteenth of its width, trained on a task small enough to train on a laptop and check by hand. The mechanism is identical, so everything the paper describes is here to be seen working. Scroll, and the camera goes where the text goes.

The picture follows the visual grammar of Brendan Bycroft's LLM Visualization [2], tensors as blocks of cells, applied to the original encoder-decoder; the code follows the structure of the Annotated Transformer [3]; the paper's own models were trained with tensor2tensor [4].

How to use this page

Scroll the article, or use ← →. Drag the stage to orbit, right-drag to pan, scroll on it to zoom; O shows the whole machine, C returns to the step's view, R replays its animation. Change the number in the bar above and every matrix is recomputed.

Go deeper: Transformer Architecture

02Overview

English numbers into German ones

The paper's task was translating news sentences: about 4.5 million English–German sentence pairs from WMT 2014, and for English–French, 36M sentences, 32,000 word-pieces. This page's model translates something smaller with the same difficulty in it. It reads an English number, , and writes the German, , joined as German writes it: .

German says the units before the tens: forty-two is zwei-und-vierzig, "two-and-forty". So the output is not a word-for-word copy of the input. To write zwei the decoder must reach past forty to the two after it. Encoder-decoder networks had learned to translate by compressing the whole sentence into one vector [5]; reordering over a long sentence is the problem attention was invented for [6], and here it can be watched happening.

The miniature has parameters against the paper's 65 million. It was trained with the paper's recipe, unchanged except for sizes, and it translates numbers it never saw in training exactly, choosing every word itself. The page computes the whole forward pass in JavaScript, checked against PyTorch to within $1 0^{- 5}$ before it was published.

03Inputs

Pieces of words, one table for both languages

A Transformer never sees letters or words. It sees integer ids for tokens drawn from a fixed vocabulary. The paper encoded English and German with byte-pair encoding into a single shared vocabulary of about 37,000 tokens [7]. Frequent words get one token; rare words split into frequent pieces, so nothing is ever out of vocabulary.

The highlighted rows of the matrix $E$ on the stage are this number's tokens. This page's vocabulary has 65 entries: three specials (<pad>, <s>, </s>), the English number words and the German pieces, with German compounds split the way a subword vocabulary would split them: drei hundert rather than dreihundert.

Because the vocabulary is shared, a single embedding matrix can serve both languages, and the paper uses one matrix three times: to embed the source, to embed the target, and, transposed, to turn the decoder's final vectors back into scores over the vocabulary. That third use is at the top of the stage; we will get there.

Go deeper: Tokenisation and BPE

04Inputs

A lookup, then a scale

Embedding is indexing: token id $i$ selects row $i$ of $E$ . The lookup is drawn on the stage as threads from the table to the encoder's input block, one per source token. The paper then multiplies by $d_{model}$ (§3.4): with $d_{model} = 32$ here that is $\times 5.657$ ; in the paper, $512 \approx 22.6$ .

x_{i}^{(0)} = d_{model} E_{[token_{i}]} + P E_{i}

The paper does not say why it scales. The usual reading, and it is an interpretation rather than the authors' statement, is about magnitudes: with tied weights the same matrix must also produce sensible logits, which keeps its entries small, while the positional encoding added next has entries of size one. Scaling lifts the token signal so position does not drown it. Hover a cell in E[x]·√32 to see the raw entry and its scaled value.

Go deeper: Input Embeddings and the Lookup Table · Weight Tying of Input and Output Embeddings

05Inputs

Where each token is, written in waves

Attention, as we are about to see, treats its input as a set: shuffle the tokens and every token's output is the same, only shuffled. Order has to be written into the vectors themselves. The paper adds a fixed pattern of sines and cosines, one frequency per pair of dimensions:

P E_{(p os, 2 i)} = sin (\frac{p os}{1000 0 ^{2 i / d_{model}}}), P E_{(p os, 2 i + 1)} = cos (\frac{p os}{1000 0 ^{2 i / d_{model}}})

The wavelengths run in a geometric progression 2π to 10000·2π. Low dimensions spin fast and distinguish neighbours; high dimensions turn slowly and tell the beginning of a long sequence from its end. On the stage, the block marked PE is that pattern for this input's positions, added cell by cell to the scaled embeddings to give $x^{(0)}$ .

Why sinusoids? The authors give two reasons. The first: for any fixed offset $k$ , $P E_{p os + k}$ is a linear function of $P E_{p os}$ , because each sine–cosine pair is a point on a circle and moving $k$ positions rotates it by a fixed angle, the same for every $p os$ :

(sin ω_{i} (p + k) cos ω_{i} (p + k)) = (cos ω_{i} k - sin ω_{i} k sin ω_{i} k cos ω_{i} k) (sin ω_{i} p cos ω_{i} p)

That is checked numerically in this page's test suite. The second reason, that sinusoids "may allow the model to extrapolate" to longer sequences than it was trained on, was offered and not tested. We test it on the research desk below. The paper also found that learned position vectors [9] did almost exactly as well.

Finally, dropout with $P_{drop} =$ 0.1 is applied to this sum during training (§5.4), and $x^{(0)}$ enters the first encoder layer.

Go deeper: Positional Encodings

06Encoder · self-attention

Three projections of every token

The encoder is a stack of identical layers, 6 in the paper and two here. Each has two sub-layers: self-attention, which lets every position gather information from every other, and a feed-forward network, which transforms each position on its own.

Self-attention begins by projecting each token's vector three ways, with three learned matrices:

Q = X W^{Q}, K = X W^{K}, V = X W^{V}

A query is what a position is looking for, a key is what it offers to be found by, and a value is what it hands over once found. On the stage each weight matrix sits directly under its output with columns aligned, so output column $j$ is the input row dotted with weight column $j$ . The animation computes one cell first: row 1 of $X$ against column 5 of $W^{Q}$ .

Each of $Q, K, V$ is drawn as four slices with gaps between them. Those are the heads. The paper uses 8 heads of width 64, so $8 \times 64 = 512$ ; here 4 heads of 8 make 32. Splitting the width keeps the cost the same as one wide head, and lets each head compute its own attention pattern.

Go deeper: Attention Mechanism · Multi-Head Attention in Depth

07Encoder · self-attention

How well does each query match each key?

For each head, every query is compared with every key by a dot product, and the result is divided by $d_{k}$ . With $n$ tokens that is an $n \times n$ table per head, the blocks labelled scores on the stage:

S = \frac{Q K ^{⊤}}{d _{k}}

The division is the "scaled" in scaled dot-product attention, and the paper explains it in a footnote. If the components of $q$ and $k$ are independent with mean 0 and variance 1, then $q \cdot k$ has mean 0 and variance $d_{k}$ . At $d_{k} = 64$ the scores spread with a standard deviation of 8, and the authors suspected that pushes the softmax into regions where it has extremely small gradients. Dividing by $d_{k}$ restores unit variance.

The figure shows the mechanism behind the suspicion. Without scaling, the largest softmax weight approaches 1 as $d_{k}$ grows and the gradient that would teach the scores, $\sum p_{j} (1 - p_{j})$ , falls away; with scaling both stay put. The paper also notes that for small $d_{k}$ the two mechanisms perform similarly. At $d_{k} = 8$ , as here, the scale should matter little. That is a prediction, and the research desk tests it.

The authors drew a further conclusion from their ablations: shrinking $d_{k}$ with everything else fixed hurts (25.1 BLEU at $d_{k} = 16$ ), which they read as a sign that "determining compatibility is not easy" and that a richer compatibility function than a dot product might help.

08Encoder · self-attention

Each row becomes a probability distribution

A softmax over each row turns scores into weights that are positive and sum to one:

a_{ij} = \frac{exp ( S _{ij} )}{\sum _{j^{'}} exp ( S _{i j^{'}} )}

On the stage, each head's softmax block now stands next to its scores, and its cells are bars: the height is the weight. Each row is one query's budget of attention, spent across the keys. Some rows put nearly everything on one token; others spread out. The pattern depends on the head, the layer and the input.

Read the inspector row by row, and switch layers. In this model the encoder's self-attention stays broad: across held-out numbers its largest weight in a row averages about one half, so each token mixes in a good deal of its neighbours. The decoder's encoder–decoder attention, which we reach later, is the opposite: nearly one-hot. Sharp attention is not the rule; it is something a head learns when its job needs it.

Go deeper: Softmax and Logits

09Encoder · self-attention

Gather, in proportion

Each head's output for position $i$ is the average of all value vectors, weighted by that row of attention:

head_{h} = softmax (\frac{Q _{h} K _{h}^{⊤}}{d _{k}}) V_{h}, o_{i} = j \sum a_{ij} v_{j}

On the stage the highlighted row of each softmax block sweeps down as the head's output, head 1 to head 4, fills in row by row. This is the only place in the whole encoder where information moves between positions. Everything else, the projections, the feed-forward network, the normalisation, acts on each position separately.

It is also why the operation parallelises so well. Every row of every head is computed at once by matrix multiplication; nothing waits for the previous token. A recurrent network has to read its input one step after another.

10Encoder · self-attention

Many small attentions, rejoined

The four head outputs are laid side by side and projected back to the model width by a fourth matrix:

MultiHead (Q, K, V) = Concat (head_{1}, \dots, head_{h}) W^{O}

Why several narrow heads instead of one wide one? In the paper's words, several heads attend "to information from different representation subspaces at different positions", while a single head has to average. Their ablations bear it out at the large end and the small end: one head of width 512 scored 24.9 BLEU on the development set against 25.8 for eight heads, and 32 heads of width 16 scored 25.4 BLEU.

Go deeper: Multi-Head Attention in Depth

11Encoder · self-attention

Add the input back, then normalise

The sub-layer's output is not passed on alone. It is added to the sub-layer's input, a residual connection [12], and the sum is normalised (§3.1):

x^{'} = LayerNorm (x + Sublayer (x)), LayerNorm (z) = γ ⊙ \frac{z - μ ( z )}{σ ( z )} + β

The violet rail on the stage is the residual path: it carries $x$ up past the whole attention station and adds it back. Layer normalisation [13] then rescales each token's vector to mean 0 and standard deviation 1 across its features, with a learned gain $γ$ and bias $β$ . For the first token of this input, the sum before normalising has . Hover the x + Attention(x) and LayerNorm blocks to see every row's statistics.

The residual path means each sub-layer only has to learn a correction to what is already there, and gradients have a short way back to early layers. It also forces every sub-layer, and the embeddings, to produce the same width, 512 in the paper.

Go deeper: Layer Normalization and Residual Connections · The Residual Stream · Normalisation: BatchNorm, LayerNorm, RMSNorm

12Encoder · feed-forward

Two matrices and a ReLU, for each token alone

The second sub-layer is a small two-layer network applied to every position separately and identically:

FFN (x) = max (0, x W_{1} + b_{1}) W_{2} + b_{2}

It widens each token from 512 to 2048 dimensions, cuts negatives to zero, and narrows back. On the stage, $W_{1}$ is the wide block, the hidden layer above it is four times the model width, and $W_{2}$ is drawn transposed so its columns line up with the hidden units. For this input, of the first layer's hidden activations are exactly zero after the ReLU: the dark cells.

The paper notes that this is "two convolutions with kernel size 1". It is also where most of the parameters are. Per layer, attention holds $4 d^{2}$ weights and the feed-forward network $2 \cdot d \cdot 4 d = 8 d^{2}$ : two thirds of every encoder layer sits in these two blocks. Then comes another residual add and LayerNorm, and the layer is done.

Go deeper: Anatomy of a Transformer Block

13Encoder · feed-forward

The same layer, again

The second encoder layer repeats the first with its own weights. The output of the last layer, highlighted at the top of the encoder, is the memory $z = (z_{1}, \dots, z_{n})$ : one vector per source token, each now informed by the whole sentence. The decoder will read it in every one of its layers.

Depth matters. With everything else fixed, the paper's development BLEU for $N = 2, 4, 8$ was 23.7, 25.3, 25.5 BLEU, against 25.8 at $N = 6$ . The gain from 6 to 8 is within noise; the loss from 6 to 2 is not.

14Decoder

The decoder reads what it has written

The decoder generates the translation one piece at a time, and at each step its input is the translation so far. During training, the true translation is fed in, shifted one place to the right behind a start token <s>, so position $i$ holds the answer to position $i - 1$ . This is teacher forcing: every position of the target is trained in parallel in one pass.

The decoder embeds its input with the same matrix $E$ (the threads now run from the right side of the table), scales by $d$ and adds the same sinusoids. On the stage its rows are <s> followed by the model's own output, because at inference that is what it reads.

15Decoder

No peeking at the future

The decoder's first sub-layer is self-attention, with one change: position $i$ may attend only to positions $j \leq i$ . The paper implements this inside the softmax: scores for illegal connections are set to $- \infty$ , so their weights come out exactly zero.

S_{ij} = {q_{i} \cdot k_{j} / d_{k} - \infty j \leq i j > i

The dark red cells above the diagonal of every score block are those $- \infty$ entries. This masking, combined with the one-position offset, guarantees that the prediction for position $i$ depends only on positions before it. That is what makes teacher forcing honest: the model cannot copy the answer it is being trained to predict, because the answer is in a masked cell.

Go deeper: Causal Masking

16Decoder

Where translation happens

The second sub-layer is where the decoder consults the source. Its queries come from the decoder; its keys and values come from the encoder's memory. The orange arcs on the stage carry the memory across to every decoder layer:

CrossAttn (Y, z) = softmax (\frac{( Y W ^{Q} ) ( z W ^{K} ) ^{⊤}}{d _{k}}) z W^{V}

Each row of these attention blocks is one German position; each column is an English token. This is the descendant of the alignment model of Bahdanau et al. [6], now one sub-layer among many.

Look at the last decoder layer. When the model is about to write the units of a German number, its attention jumps forward in the English to the units word, then back to the tens. The reordering is visible as a step out of the diagonal. Whether that is the rule rather than a lucky example is a question the research desk answers.

17Decoder

The decoder layer, complete

A decoder layer is masked self-attention, encoder–decoder attention and a feed-forward network, each wrapped in a residual connection and LayerNorm. Three sub-layers where the encoder has two, which is why the decoder tower on the stage is taller. The paper stacks six; this page stacks two. A decoder layer holds $2 \times 4 d^{2}$ attention weights and $8 d^{2}$ in its feed-forward network.

18Output

From vectors back to words

The decoder's final vectors are turned into a score for every vocabulary entry by a linear map, and a softmax turns scores into probabilities. The paper shares this map with the embeddings: the output layer is $E^{⊤}$ , the same matrix that sits at the bottom of the stage. The long teal line joins the two.

P (y_{i + 1} ∣ y_{\leq i}, x) = softmax (h_{i} E^{⊤})

Tying, introduced for language models by Press and Wolf [15], saves a vocabulary-sized matrix (about 19 million weights at the paper's size) and makes a word's input vector and its output vector the same object. The tallest bar in each row of the probability block is the model's choice. This model is confident: its least certain greedy choice for this input still had probability.

Go deeper: Weight Tying of Input and Output Embeddings · Softmax and Logits

19Output

Autoregression, and the beam

At inference there is no true translation to feed in. The decoder starts from <s>, predicts one piece, appends it to its own input (the yellow loop on the stage), and runs again, until it predicts </s> or reaches the limit of input length + 50. The encoder runs once; the decoder runs once per output piece.

Always taking the most probable piece ("greedy" decoding) can paint a translation into a corner. The paper keeps the four best partial translations at each step instead, a beam search of beam 4, α = 0.6, and ranks finished ones with the length penalty of Google's neural translation system [16]:

score (Y) = \frac{lo g P ( Y ∣ X )}{lp ( Y )}, lp (Y) = \frac{( 5 + ∣ Y ∣ ) ^{α}}{( 5 + 1 ) ^{α}}

Without the penalty, every extra word lowers $lo g P$ and the search prefers short outputs. The page runs the same beam search on the live input:

Go deeper: Autoregressive Generation · Sampling and Decoding

20Training

What the model is trained to do

Training minimises cross-entropy between the predicted distribution and the true next piece, at every target position in parallel. The paper softens the target with label smoothing, ε_ls = 0.1 [17]: instead of probability 1 on the right answer, the target keeps $1 - ε$ there and spreads $ε$ over the vocabulary.

q^{'} (k) = (1 - ε) 1 [k = y] + \frac{ε}{V}

The authors say plainly that this "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score". Their ablation shows both halves: without it, development perplexity improved from 4.92 to perplexity 4.67, 25.3 BLEU, while BLEU fell from 25.8. Later analysis traced the effect to the geometry it imposes on the penultimate layer, pulling examples of each class into tight, equidistant clusters [18].

Go deeper: Next-Token Prediction and Cross-Entropy Loss

21Training

A learning rate that rises, then decays

The optimiser is Adam [19] with β1 = 0.9, β2 = 0.98, ε = 10⁻⁹. Note $β_{2} = 0.98$ rather than the usual 0.999: a shorter memory for the squared-gradient estimate. The learning rate follows a formula that has become famous in its own right:

lrate = d_{model}^{- 0.5} \cdot min (step^{- 0.5}, step \cdot warmup^{- 1.5})

It climbs linearly for 4,000 steps and then decays with the inverse square root of the step. At $d_{model} = 512$ the peak is $7.0 \times 1 0^{- 4}$ , reached at step 4,000. Post-LN Transformers are known to diverge without such a warm-up [14].

The scale of the original run: batches of about 25,000 source + 25,000 target tokens, 8 NVIDIA P100 GPUs, 100,000 steps, 12 hours, 0.4 s a step for the base model and 300,000 steps, 3.5 days, 1.0 s a step for the big one, whose dropout was raised to 0.3. This page's miniature followed the same recipe on a laptop CPU:

Go deeper: Optimisers: SGD, Adam, AdamW, Lion

22Training

Noise in training, an average at the end

Dropout [20] zeroes a random tenth of the values at two kinds of places: each sub-layer's output before it is added to the residual stream, and the sum of embeddings and positional encodings. The stage shows one such mask on the first attention output. Removing dropout cost the base model more than a BLEU point: 24.6 BLEU against 25.8.

At the end, the paper does not use the last checkpoint. It averages the weights of the last 5 (base), last 20 (big), a cheap ensemble in weight space.

Go deeper: Dropout and Modern Regularisation

23Why attention

Why not recurrence?

The paper's §4 argues from three measures: total computation per layer, how much of it must happen in sequence, and the length of the path a signal travels between two positions. Long paths make long-range dependencies hard to learn [21].

Layer	Cost per layer	Sequential steps	Longest path
Self-attention	O(n²·d)	O(1)	O(1)
Recurrent	O(n·d²)	O(n)	O(n)
Convolutional	O(k·n·d²)	O(1)	O(log_k n)
Self-attention, restricted to r	O(r·n·d)	O(1)	O(n/r)

Every pair of positions is one attention step apart. Self-attention is cheaper than recurrence when the sequence is shorter than the vector width, which for sentences of a few dozen subwords and $d = 512$ it almost always was. The quadratic term is the price, and it is why long-context work since has been largely about attention's cost.

Go deeper: Recurrent Networks: RNN, LSTM, GRU · The FLOPs of a Transformer Forward Pass

24Why attention

Better, and cheaper to train

Translation quality is measured in BLEU, the overlap of n-grams between the output and human references [22]. On WMT 2014 English→German the big model scored 28.4, above the best previous result of 26.36, which was an ensemble; the base model scored 27.3 (and 38.1 on English→French). On English→French the big model scored 41.8.

Model	EN–DE	EN–FR	Training FLOPs
GNMT + RL	24.6	39.92	2.3·10¹⁹ / 1.4·10²⁰
ConvS2S	25.16	40.46	9.6·10¹⁸ / 1.5·10²⁰
ConvS2S ensemble	26.36	41.29	7.7·10¹⁹ / 1.2·10²¹
Transformer, base	27.3	38.1	3.3·10¹⁸
Transformer, big	28.4	41.8	2.3·10¹⁹

The cost column is an estimate: training time × GPUs × sustained throughput. The base Transformer's 3.3·10¹⁸ FLOPs is a fraction of any competitor's; even the big model's 2.3·10¹⁹ FLOPs is below every English→French competitor and level with GNMT's English→German run.

The architecture also transferred: a 4-layer Transformer parsed English sentences into constituency trees at 91.3 (WSJ only), 92.7 (semi-supervised) F1 on WSJ section 23, with 4 layers, d_model 1024, about 40K sentences and little tuning.

25Why attention

Counting to 65 million

With every block on the stage, the parameter count can be done by hand. Per encoder layer: four $d \times d$ attention matrices with biases, the two feed-forward matrices with biases, two LayerNorms. Per decoder layer: twice the attention, plus three LayerNorms. Plus one shared $V \times d$ embedding. The highlighted blocks on the stage are the parameters; everything else is computed.

At a vocabulary of exactly 37,000 the formula gives 63.08 million for base and 214.25 million for big, against the paper's 65 million and 213 million. The big model matches to within half a percent; the base model is 2 million short, about what a vocabulary of 41,000 rather than "about 37000" would add. The paper does not give the exact vocabulary size, so the gap cannot be closed from the text alone.

Go deeper: Counting Transformer Parameters

26Research desk

Do the paper's ablations hold at 1/16 scale?

The paper's Table 3 varies one design choice at a time. I re-ran the ones that make sense on this task, at miniature scale, with the hypotheses written down before any run. Each variant trained from scratch with the recipe above for 2,500 steps, with two seeds, and was scored on 2,000 held-out numbers by exact match under free-running greedy decoding.

27Research desk

Testing a conjecture the paper left open

Claim

Sinusoidal encodings "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training" (§3.5). The paper offers this as the reason for choosing them and does not test it.

Hypothesis

A model trained only on inputs of at most six English words will translate seven- to nine-word inputs if it uses sinusoids, and will not if it uses learned position vectors, whose entries for positions beyond six never receive a gradient.

Design

Same recipe, three seeds per arm. Training draws only numbers whose English has ≤ 6 words. Every piece of vocabulary and every grammatical pattern still appears in training; only the total length is new. Scored per input length.

The broader literature agrees with the cautious reading. Language models with sinusoidal encodings degrade quickly past their training length [23], which motivated relative schemes such as relative position representations [24], rotary embeddings [25] and linear biases [23]; and in decoder-only models even no positional encoding can generalise better than the original [26].

28Research desk

Does attention learn which word translates which?

Question

Encoder–decoder attention descends from an explicit alignment model [6]. Does a Transformer trained only to translate learn to align words, including across reordering, or does it spread attention in ways that only work collectively?

Hypothesis

At least one last-layer cross-attention head points at the English word being translated for most German pieces, including the reordered units, well above a baseline that attends by relative position alone.

Measure

For each German piece with an English counterpart, take the decoder row that emits it and check whether each head's attention peaks on that counterpart. Gold alignments follow from the grammar; "und" has none.

29Research desk

What this suggests, and what to test next

Each of these can be run with the scripts in this page's research/ directory:

Which heads carry the reordering? Ablate each last-layer cross-attention head at inference and measure exact match on numbers with and without "und". If one head is necessary for reordering and not for monotone pieces, the alignment found above is causal, not incidental.
Extrapolation with relative positions. Repeat the length test with rotary embeddings [25] or linear biases [23] on the same split. The miniature makes a three-way comparison cheap enough to run with ten seeds.
Does post-LN need warm-up here? Train with warm-up 0 and with pre-LN [14]. At this width the instability may not appear at all, which would locate it in scale rather than in the architecture.
Compositional generalisation. Hold out every number containing a particular pattern (say, any "sieben und …") and test whether the model composes it from parts seen elsewhere. Transformers are known to struggle with this kind of systematic recombination as problems grow [30]; numbers give an exact, graded test.
Is d_k = 8 too small to matter? The scaling result predicts a crossover. Sweep $d_{k}$ from 2 to 64 at fixed width, with and without the $1/ d_{k}$ , and find where it starts to matter.

The miniature, its weights and every experiment are in the page's directory; the same JavaScript that draws the stage is what the tests check against PyTorch. Corrections and replications are welcome.

References

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017); arXiv:1706.03762v7 The subject of this page. Quotes are from the v7 LaTeX source.
Bycroft, B. (2023). LLM Visualization. bbycroft.net The visual grammar of tensors as blocks of cells that this page follows.
Rush, A. M. (2018). The Annotated Transformer. Workshop for NLP Open Source Software (NLP-OSS); updated 2022 by Huang, Subramanian, Sum, Almubarak and Biderman
Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A. N. et al. (2018). Tensor2Tensor for Neural Machine Translation. AMTA 2018; arXiv:1803.07416 The library the paper's models were trained with.
Sutskever, I., Vinyals, O. and Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NIPS 2014; arXiv:1409.3215
Bahdanau, D., Cho, K. and Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015; arXiv:1409.0473
Sennrich, R., Haddow, B. and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016; arXiv:1508.07909 Byte-pair encoding for NMT.
Britz, D., Goldie, A., Luong, M.-T. and Le, Q. (2017). Massive Exploration of Neural Machine Translation Architectures. EMNLP 2017; arXiv:1703.03906
Gehring, J., Auli, M., Grangier, D., Yarats, D. and Dauphin, Y. N. (2017). Convolutional Sequence to Sequence Learning. ICML 2017; arXiv:1705.03122 ConvS2S; learned position embeddings.
Michel, P., Levy, O. and Neubig, G. (2019). Are Sixteen Heads Really Better than One?. NeurIPS 2019; arXiv:1905.10650
Voita, E., Talbot, D., Moiseev, F., Sennrich, R. and Titov, I. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019; arXiv:1905.09418
He, K., Zhang, X., Ren, S. and Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016; arXiv:1512.03385
Ba, J. L., Kiros, J. R. and Hinton, G. E. (2016). Layer Normalization. arXiv:1607.06450
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L. and Liu, T.-Y. (2020). On Layer Normalization in the Transformer Architecture. ICML 2020; arXiv:2002.04745
Press, O. and Wolf, L. (2017). Using the Output Embedding to Improve Language Models. EACL 2017; arXiv:1608.05859
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M. et al. (2016). Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144 Word-pieces; the beam-search length penalty.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. CVPR 2016; arXiv:1512.00567 Introduces label smoothing.
Müller, R., Kornblith, S. and Hinton, G. (2019). When Does Label Smoothing Help?. NeurIPS 2019; arXiv:1906.02629
Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015; arXiv:1412.6980
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15
Hochreiter, S., Bengio, Y., Frasconi, P. and Schmidhuber, J. (2001). Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies. A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press
Papineni, K., Roukos, S., Ward, T. and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL 2002
Press, O., Smith, N. A. and Lewis, M. (2022). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR 2022; arXiv:2108.12409
Shaw, P., Uszkoreit, J. and Vaswani, A. (2018). Self-Attention with Relative Position Representations. NAACL 2018; arXiv:1803.02155
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B. and Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864
Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P. and Reddy, S. (2023). The Impact of Positional Encoding on Length Generalization in Transformers. NeurIPS 2023; arXiv:2305.19466
Jain, S. and Wallace, B. C. (2019). Attention is not Explanation. NAACL 2019; arXiv:1902.10186
Wiegreffe, S. and Pinter, Y. (2019). Attention is not not Explanation. EMNLP 2019; arXiv:1908.04626
Garg, S., Peitz, S., Nallasamy, U. and Paulik, M. (2019). Jointly Learning to Align and Translate with Transformer Models. EMNLP 2019; arXiv:1909.02074
Dziri, N., Lu, X., Sclar, M. et al. (2023). Faith and Fate: Limits of Transformers on Compositionality. NeurIPS 2023; arXiv:2305.18654

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
about 4.5 million	WMT 2014 English-German training pairs	[1] §5.1 We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs.
36M sentences, 32,000 word-pieces	WMT 2014 English-French	[1] §5.1 For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary.
65 million	parameters, base	[1] Table 3 base & 6 & 512 & 2048 & 8 & 64 & 64 & 0.1 & 0.1 & 100K & 4.92 & 25.8 & 65
about 37,000	shared BPE vocabulary	[1] §5.1 Sentences were encoded using byte-pair encoding, which has a shared source-target vocabulary of about 37000 tokens.
reference [3]	the BPE citation	[1] §5.1 Sentences were encoded using byte-pair encoding [3]
× √d_model	embedding scale	[1] §3.4 In the embedding layers, we multiply those weights by √d_model.
2π to 10000·2π	wavelengths	[1] §3.5 The wavelengths form a geometric progression from 2π to 10000 · 2π.
relative offsets are linear maps	why sinusoids (1)	[1] §3.5 We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE_{pos+k} can be represented as a linear function of PE_{pos}.
may extrapolate	why sinusoids (2)	[1] §3.5 We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training.
25.7 BLEU	learned positions instead of sinusoids	[1] Table 3 (E) In row (E) we replace our sinusoidal positional encoding with learned positional embeddings, and observe nearly identical results to the base model.
0.1	residual dropout, base	[1] §5.4 For the base model, we use a rate of P_drop=0.1.
each sub-layer output, and the embedding sums	where dropout goes	[1] §5.4 We apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks.
6	layers in each stack	[1] §3.1 The encoder is composed of a stack of N=6 identical layers.
8	attention heads	[1] §3.2.2 In this work we employ h=8 parallel attention layers, or heads.
64	per-head key and value width	[1] §3.2.2 For each of these we use d_k=d_v=d_model/h=64.
variance d_k	why scale by 1/√d_k	[1] §3.2.1, footnote 4 assume that the components of q and k are independent random variables with mean 0 and variance 1. Then their dot product, q·k = Σ q_i k_i, has mean 0 and variance d_k.
vanishing softmax gradients	the suspected failure	[1] §3.2.1 We suspect that for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.
similar	small d_k	[1] §3.2.1 While for small values of d_k the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of d_k.
25.1 BLEU at d_k = 16	smaller d_k	[1] Table 3 (B) we observe that reducing the attention key size d_k hurts model quality. This suggests that determining compatibility is not easy and that a more sophisticated compatibility function than dot product may be beneficial.
different subspaces	why several heads	[1] §3.2.2 Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
24.9 BLEU	one head (d_k = 512)	[1] Table 3 (A) While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.
25.8	dev BLEU, base (newstest2013)	[1] Table 3 base & ... & 4.92 & 25.8 & 65
25.4 BLEU	32 heads (d_k = 16)	[1] Table 3 (A) & & & & 32 & 16 & 16 & & & & 5.01 & 25.4 &
LayerNorm(x + Sublayer(x))	sub-layer output	[1] §3.1 That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself.
512	model width	[1] §3.1 all sub-layers in the model, as well as the embedding layers, produce outputs of dimension d_model=512.
2048	feed-forward inner width	[1] §3.3 The dimensionality of input and output is d_model=512, and the inner-layer has dimensionality d_ff=2048.
kernel size 1	the FFN as convolutions	[1] §3.3 Another way of describing this is as two convolutions with kernel size 1.
23.7, 25.3, 25.5 BLEU	depth N = 2, 4, 8	[1] Table 3 (C) & 2 & ... & 6.11 & 23.7 & 36 \ & 4 & ... & 5.19 & 25.3 & 50 \ & 8 & ... & 4.88 & 25.5 & 80
−∞ before the softmax	decoder mask	[1] §3.2.3 We implement this inside of scaled dot-product attention by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections.
shifted right	outputs offset by one	[1] §3.1 This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.
one matrix, three uses	weight sharing	[1] §3.4 In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation.
input length + 50	longest output	[1] §6.1 We set the maximum output length during inference to input length + 50, but terminate early when possible.
beam 4, α = 0.6	beam search	[1] §6.1 We used beam search with a beam size of 4 and length penalty α=0.6.
lp(Y) = (5+\|Y\|)^α / (5+1)^α	length penalty	[16] §7 lp(Y) = (5 + \|Y\|)^α / (5 + 1)^α
ε_ls = 0.1	label smoothing	[1] §5.4 During training, we employed label smoothing of value ε_ls=0.1. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
perplexity 4.67, 25.3 BLEU	no label smoothing	[1] Table 3 (D) & & & & & & & & 0.0 & & 4.67 & 25.3 &
β1 = 0.9, β2 = 0.98, ε = 10⁻⁹	Adam settings	[1] §5.3 We used the Adam optimizer with β1=0.9, β2=0.98 and ε=10^-9.
4,000	warm-up steps	[1] §5.3 We used warmup_steps=4000.
about 25,000 source + 25,000 target	tokens per batch	[1] §5.1 Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens.
8 NVIDIA P100 GPUs	hardware	[1] §5.2 We trained our models on one machine with 8 NVIDIA P100 GPUs.
100,000 steps, 12 hours, 0.4 s a step	base model training	[1] §5.2 For our base models using the hyperparameters described throughout the paper, each training step took about 0.4 seconds. We trained the base models for a total of 100,000 steps or 12 hours.
300,000 steps, 3.5 days, 1.0 s a step	big model training	[1] §5.2 For our big models, (described on the bottom line of table 3), step time was 1.0 seconds. The big models were trained for 300,000 steps (3.5 days).
24.6 BLEU	no dropout	[1] Table 3 (D) & & & & & & & 0.0 & & & 5.77 & 24.6 &
last 5 (base), last 20 (big)	checkpoint averaging	[1] §6.1 For the base models, we used a single model obtained by averaging the last 5 checkpoints, which were written at 10-minute intervals. For the big models, we averaged the last 20 checkpoints.
three types	regularisers announced	[1] §5.4 We employ three types of regularization during training:
n < d	when attention is cheaper	[1] §4 self-attention layers are faster than recurrent layers when the sequence length n is smaller than the representation dimensionality d
28.4	EN→DE BLEU, big	[1] Table 2, §6.1 establishing a new state-of-the-art BLEU score of 28.4.
26.36	previous best EN→DE (ensemble)	[1] Table 2 ConvS2S Ensemble & 26.36 & 41.29
27.3	EN→DE BLEU, base	[1] Table 2 Transformer (base model) & 27.3 & 38.1
38.1	EN→FR BLEU, base	[1] Table 2 Transformer (base model) & 27.3 & 38.1
41.8	EN→FR BLEU, big (Table 2 and abstract)	[1] abstract, Table 2 our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs
time × GPUs × 9.5 TFLOPS (P100)	how FLOPs were estimated	[1] §6.1, footnote We estimate the number of floating point operations used to train a model by multiplying the training time, the number of GPUs used, and an estimate of the sustained single-precision floating-point capacity of each GPU.
3.3·10¹⁸ FLOPs	training cost, base	[1] Table 2 Transformer (base model) ... 3.3·10^18
2.3·10¹⁹ FLOPs	training cost, big	[1] Table 2 Transformer (big) ... 2.3·10^19
41.0	EN→FR BLEU, big (body text)	[1] §6.1 On the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0
91.3 (WSJ only), 92.7 (semi-supervised)	WSJ 23 F1	[1] Table 4 Transformer (4 layers) & WSJ only, discriminative & 91.3 ... Transformer (4 layers) & semi-supervised & 92.7
4 layers, d_model 1024, about 40K sentences	parsing setup	[1] §6.3 We trained a 4-layer transformer with d_model = 1024 on the Wall Street Journal (WSJ) portion of the Penn Treebank, about 40K training sentences.
213 million	parameters, big	[1] Table 3 big & 6 & 1024 & 4096 & 16 & & & 0.3 & & 300K & 4.33 & 26.4 & 213

How this was made

Everything on the stage is computed in the page from trained weights: a 61,472-parameter encoder-decoder with the paper's architecture and recipe (post-LN, sinusoidal positions, three-way tied embeddings, Adam 0.9/0.98/1e-9 with the warm-up formula, dropout 0.1, label smoothing 0.1, beam 4 with alpha 0.6), trained in PyTorch for 4,000 steps on a CPU. Its JavaScript forward pass, greedy and beam search match PyTorch to within 1e-5 on reference inputs (test.mjs, 66 checks), and the same tests recompute every derived number in the text: parameter counts, the learning-rate peak, the positional-encoding rotation identity and the variance of q·k. Paper figures are quoted from the arXiv v7 LaTeX source and listed in the page's fact ledger. The research-desk experiments are reproducible from research/.

Built with the dissection skill. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization; the code structure to Rush's Annotated Transformer. Source, weights, training scripts and experiments: anatomy/ai-ml/2026-10-15-transformer/ in the site's repository.

Cite this page

@misc{sigmoid-transformer,
  title  = {Anatomy of the Transformer},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/transformer}
}