Dissection · Belrose et al., 2023 · nostalgebraist, 2020

Where the answer forms

A real 0.5B language model read mid-computation by two lenses, the model's own last step and one trained per layer, and then changed by patching to see where the answer is actually made.

PapersBelrose et al. 2023 (tuned lens); nostalgebraist 2020 (logit lens)

ModelQwen2.5-0.5B-Instruct, 494M parameters, real weights

Reading26 min · 23 steps

Published2026-10-19

01The question

What is a language model holding halfway through its computation?

A language model turns a prompt into a probability for every possible next token by passing one vector per position through a stack of blocks. After block 12 of 24 there is a vector, but no probabilities yet. Two methods turn it into probabilities anyway: the logit lens reads it with the model's own last step [1], and the tuned lens first moves it with a map trained for that layer [2].

The logit lens rests on one observation: applying the same function to the activations of intermediate layers gives distributions that make intuitive sense. The tuned lens is its repair: it trains one affine map per layer so that the decoded hidden state matches the final layer's logits as closely as possible. Its authors' headline claim is that tuned-lens predictions have substantially lower perplexity than logit-lens predictions and are more representative of the final layer's distribution, and their description of the older method is that it is "often brittle".

The stage shows one real model, Qwen2.5-0.5B-Instruct, at the one position where it is about to write a token: a column of its 25 internal states, the two lenses' reading of each, and what happens to the answer when parts of the model are swapped for the same parts from a different prompt. The prompt on the stage is , the gold answer is , and the model's own top choice at the last layer is .

This piece asks three questions of that model, written down before any run: is the tuned lens really closer to what the model computes; at what depth does each lens find the answer for facts, copies and sums; and when we change the model by patching, does the answer form where the lenses say it does? The first is a replication, the second and third are not in the original paper.

Only the first held. The tuned lens is far closer to the model's final distribution than the logit lens, yet on these 90 prompts it does not find the answer earlier or name the model's decisive blocks any better. Both lenses read the answer about when the last position's stream commits to it, give or take two layers, except for sums, where both read it four or five layers late.

Go deeper: Mechanistic Interpretability Primer · Unembedding and the Logit Lens

02The question

Which model is being read, and at what size?

This is not a miniature. It is the released model: 24 layers, 14 query heads sharing 2 key-value heads, tied embeddings [3], that is 24 blocks with 14 query heads over 2 key-value heads in each. The architecture is grouped-query attention, SwiGLU, rotary positions, QKV bias and pre-normalised RMSNorm, and its sizes come from config.json at revision 7ae5576 [4]: a residual width of 896, an MLP width of 4,864, and 151,936 embedding rows that double as the output layer (tied).

The block drawn on the far left of the stage is schematic and to scale: one cell is 32 hidden dimensions, so the residual stream is 28 cells wide, the 14 query heads are 14 groups of 2 cells, the two key heads and the two value heads are two groups of 2 cells each, and the MLP's hidden layer is 152 cells, … parameters in all (24 such blocks and the tied embedding). It carries no data; the data are on the residual stream to its right.

Go deeper: Transformer Architecture · Multi-Query and Grouped-Query Attention · Normalisation

03The stream

Which position do we read?

A model that writes the next token reads its answer from the last position of the prompt, so that is the one drawn. The study uses three kinds of one-token answer: a fact ("The capital of France is" → Paris, or an official language), a copy (a four-word list repeated one word short, so the model must write the fourth word again), and a sum ("7 − 5 = " → 2). The stage holds nine of the 90 study prompts, the first of each of nine prompt sets, chosen by position in a seeded list and not by how they look.

Now showing: , a , with gold answer . Change the prompt with the box above the stage. For the patching checks each prompt has a corrupt partner with the same template and a different answer: here with answer .

Go deeper: Tokenisation and Arithmetic · Byte-Pair Encoding

04The stream

What is the residual stream, and what does it look like?

Every block reads the stream and adds to it. In the notation of the tuned-lens paper, each of attention and the MLP reads its input from the stream with a linear projection and writes back by adding a linear projection [5]:

h_{ℓ + 1} = h_{ℓ} + F_{ℓ} (h_{ℓ})

where $h_{ℓ} \in R^{896}$ is the stream after $ℓ$ blocks and $F_{ℓ}$ is what block $ℓ + 1$ adds. So there are 25 states of the same vector, from the token embedding ( $ℓ = 0$ ) to the vector the output layer reads ( $ℓ = 24$ ). On the stage each is a tile of 8 × 112 cells, one cell per dimension; blue is negative, amber is positive, height is magnitude, all on one scale.

Most cells are small and a few are huge, which is why the tiles look like dark plains with isolated towers. Large dimensions of this kind were reported as massive activations, some 100,000 times larger than the others [6], and the first token of a text acts as an attention sink [7]. Measured on ten held-out windows, the first token's stream has norm about 1,630 at layers 6, 12 and 18, against medians of 12, 16 and 30 for all the other positions, which is why it is left out of the lens training and evaluation below.

Go deeper: The Residual Stream · Residual Connections · Attention Sinks

05The stream

How much does one block change the stream?

The left-hand bars on the stage show, for each block, how large its attention write and its MLP write are relative to the stream they are added to. The Geva et al. reading of this is that the output of each feed-forward layer is an additive update to a distribution over the vocabulary [8]; the residual-network version is that residual connections encourage blocks to move along the negative gradient of the loss [9]. If both are right, the middle of the stream should change gradually, and the model should tolerate losing a middle block: on Pythia 6.9B, only the very first layer was crucial, and every other single-layer ablation raised perplexity almost imperceptibly [2]. More broadly, models are remarkably robust to dropping middle layers [10], and the last two of their four proposed stages are prediction ensembling and residual sharpening, which would put most of the work of forming an answer in the final blocks.

The size of a write says nothing about whether it matters to the answer. The patching steps below measure that directly.

Go deeper: Residual Connections · Knowledge Distillation

06The logit lens

How does a stream become a distribution?

At the end of the stack the stream is normalised and multiplied by the unembedding matrix, one row per vocabulary token:

z = W_{U} RMSNorm (h_{24}), p = softmax (z)

RMSNorm (h)_{i} = γ_{i} \frac{h _{i}}{\frac{1}{d} \sum _{j} h _{j}^{2} + ϵ}

with $d = 896$ , learned gains $γ$ and $ϵ = 1 0^{- 6}$ (1e-6). Qwen2.5-0.5B ties $W_{U}$ to the input embedding, so the same 151,936 × 896 matrix that turns a token into a vector at layer 0 turns a vector into logits at layer 24. On the stage the plain slab above the last tile stands for it (it would be 19,000 cells tall at this scale, so it is drawn six). Above it is the model's own distribution at this position: .

This is the only place the stream is ever read as words. Everything before it is a vector in a basis the model keeps to itself, and the lens question is whether that basis is close enough to the last layer's for the same decoder to work earlier.

Go deeper: Unembedding and the Logit Lens · Probability and Information Theory

07The logit lens

What if the model stopped here and answered?

The logit lens answers by applying the last step to an earlier state, pretending the remaining blocks write nothing:

LL_{ℓ} (h_{ℓ}) = softmax (W_{U} RMSNorm (h_{ℓ}))

At $ℓ = 24$ this is exactly the model, and the test suite checks that the page reproduces the recorded output. At $ℓ < 24$ it is a guess about what the model would say if it stopped. The left station on the stage is computed from the recorded residual in your browser and moves with the slider below.

The page reads the full 151,936-way softmax without holding 136 million weights: the stored residual goes through RMSNorm, is multiplied by the unembedding rows of 1,500 candidate tokens (the top eight of every recorded reading, the gold and corrupt answers, and the most frequent tokens of the lens training text), and the full-vocabulary log normaliser, recorded once per layer, turns the candidate logits into exact log-probabilities. The test suite checks the result against the model's own readings of the same layers.

Go deeper: Unembedding and the Logit Lens · Mechanistic Interpretability Primer

08The logit lens

Does the logit lens show a sensible guess in the middle?

The original post found that for GPT-2 the readings of the middle layers are sensible: at some point in the middle the model has formed a "pretty good guess" as to the next token and later layers refine it [1]. A later edit to the post added a variant of the decoder that dramatically helps with interpreting some models, the first sign that the plain version fails on some of them. It also reported that only the inputs look like the input tokens and, on why that is odd, that the input space immediately vanishes and the stream is in output space the whole way. The tuned-lens paper found a different pathology in other models: in BLOOM and OPT 125M the logit lens's top prediction is often the input token, in more than half the layers [2].

Go deeper: Unembedding and the Logit Lens · Superposition and Polysemantic Neurons

09The logit lens

Why would reading the raw stream be a poor decoder?

The tuned-lens paper gives two reasons. The first is drift: the covariance matrices of hidden states at different layers drift apart as more layers separate them, so a decoder tuned to the last layer's basis may misread earlier ones. The second is bias: even when the logit lens looks sensible it is off by around 4 to 5 bits for most layers in the marginal distribution it assigns to the vocabulary, on GPT-Neo-2.7B. A third is visible on the stage, where the callout marks the largest dimension of the stream at layer 12. A decoder that multiplies the whole normalised vector by one fixed matrix gives such a dimension a large say in every logit, whether or not it carries the answer.

The amount by which the logit lens misses the model's distribution is measured below on held-out text, for every layer.

Go deeper: Superposition and Polysemantic Neurons · Attention Sinks

10The tuned lens

What does a translator do?

The repair is to learn, for each layer, a map from that layer's basis to the one the output layer expects. The paper's "tuned lens formula" is a learned affine map in front of the logit lens [2]. This page writes it in residual form, so that the starting point is the logit lens itself:

TL_{ℓ} (h_{ℓ}) = softmax (W_{U} RMSNorm (h_{ℓ} + A_{ℓ} h_{ℓ} + b_{ℓ}))

Here $A_{ℓ}$ is an $896 \times 896$ matrix and $b_{ℓ}$ a vector of 896 numbers. With $A_{ℓ} = 0$ and $b_{ℓ} = 0$ this is the logit lens, and the paper starts every translator at the identity for the same reason. Each translator has 803,712 parameters (an 896 × 896 matrix and a bias vector), 24 of them in all: 19.3 million, 3.9% of the model's own size. The loss that trains it is the model's own opinion, a distillation of the final layer's logits:

L_{ℓ} = E KL (p_{final} ∥ TL_{ℓ} (h_{ℓ}))

Because the target is the model's own distribution and not the true next token, the lens is not rewarded for knowing more than the model. A concurrent paper used the same idea, linear transformations from hidden layers to the last layer, and likewise found its predictions more accurate than reading the layers directly [11].

The tile on the left of the stage is the translator for layer 12, mean-pooled from 896 × 896 to 56 × 56 for display; the slider in the figure under "Stop early and decode" can switch the page to the tuned lens at that layer, where the page applies the full stored matrix to the stored residual.

Go deeper: Knowledge Distillation · Probability and Information Theory

11The tuned lens

How were the translators trained here?

The recipe is the paper's with the budget cut to fit a CPU without a GPU. The original trained with SGD with Nesterov momentum and a linear decay over 250 steps and batches of 2^18 tokens, on a slice of the pre-training validation set, taking under an hour on a node of eight data-centre GPUs. This page trained on WikiText-2 (freely available) [12]:

	Belrose et al.	this page
model	GPT-2, GPT-Neo, Pythia, OPT, BLOOM	Qwen2.5-0.5B-Instruct
optimiser	SGD with Nesterov momentum (Muon recommended later)	Muon on the matrices, Adam on the biases
steps × tokens	250 × 2¹⁸ ≈ 65 million	300 × 768 = 230,400 (91,800 distinct)
data	pre-training validation text	360 WikiText-2 windows of 256 tokens (40 more for the optimiser pilot)
vocabulary in the loss	full	8,192 most frequent training tokens, renormalised
hardware and time	8 × A40, under an hour	3 CPU threads, about 21 minutes for 24 translators

Two choices need defending. First, the full unembedding is 151,936 wide, and with it each of the 24 layers costs twenty times more per step on a CPU; the loss uses the 8,192 most frequent tokens and renormalises both distributions over them, while every evaluation on this page uses the full vocabulary. Second, the first token of each window is dropped, for the reason on the stream step. Muon is an optimizer for 2D parameters of neural network hidden layers [13], which suits a translator matrix. The optimiser was chosen on the last 40 training windows by a pilot, never on the evaluation text: Muon at learning rate 0.06 beat Adam and Muon at 0.03 by the pilot's mean KL, and plain SGD diverged at the paper's learning rate, which was set for batches about 340 times larger.

Go deeper: Optimisers: Adam, AdamW, Lion · Knowledge Distillation

12The tuned lens

How far is each lens from what the model finally says?

The measure is the KL divergence, in bits, from the model's own final distribution to the lens's, averaged over positions of held-out text the translators never saw. Zero would mean the lens already says what the model will say.

The pre-registered test of this, with its verdict, is the first experiment on the research desk.

The same data in the other two tabs: the cross-entropy against the true next word, which is the "perplexity" of the paper's plots, and the share of positions where the lens and the model agree on the single most likely token.

Go deeper: Probability and Information Theory · Unembedding and the Logit Lens

13Reading the answer

At which layer does a lens first hold the answer for good?

The tuned-lens paper defines the prediction depth of a prompt as the number of layers after which the model's top-1 prediction stops changing and finds it correlates with how late an example is learned in training. This page uses a stricter version, since the question is about a known answer $a$ :

D_{lens} = min {ℓ : ar g max Lens_{ℓ^{'}} (h_{ℓ^{'}}) = a for every ℓ^{'} \geq ℓ}

The smallest layer from which the gold token is the lens's top token at every later layer too, up to the output where both lenses coincide with the model. On the stage the layers from that depth onward are lit in each lens's column.

Go deeper: Unembedding and the Logit Lens · Knowledge Distillation

14Reading the answer

Does the depth depend on what the model is doing?

Three families, thirty prompts each in three disjoint sets of ten. For a fact the answer lives in the weights and has to be fetched from the subject token; for a copy it sits in the context and has to be found by attention (the induction pattern of a repeated list); for a sum it has to be computed from two single-digit operands, and Qwen writes every digit as its own token, so the answer is one digit token. The original post saw copying resolved late: for a repeated word, the model predicted it correctly at the last position but only in the very last layers [1].

The pre-registered comparison of the two lenses on these prompts, with its verdict, is the second experiment on the research desk.

Go deeper: Induction Heads and In-Context Learning · Tokenisation and Arithmetic · Attention

15Checking by patching

How do we check a lens against the model itself?

A lens is a reading. To test a reading, change the thing read and see whether the model behaves as the reading says. The tool is activation patching in its noising direction: run the clean prompt, but replace one internal state with the same state from a different prompt, and see what the patch breaks, which tells us the activations were necessary to maintain for the behaviour [14]. The corrupt prompt must be chosen with care, since causal tracing depends on a corruption rule [15]; here it is another prompt with the same template and a different answer, such as against .

The measure is the logit difference between the two answers, which is the metric those authors trust the most:

LD = z_{a} - z_{a^{'}}

s_{ℓ} = \frac{LD _{clean} - LD _{patched, ℓ}}{LD _{clean} - LD _{corrupt}}

where $a$ is the clean answer, $a^{'}$ the corrupt prompt's, and $s_{ℓ} = 0$ means the model still behaves as on the clean prompt, $s_{ℓ} = 1$ that it behaves as on the corrupt one. Two patches are used. A residual patch overwrites the last position's stream after $ℓ$ blocks with the corrupt run's. A block patch overwrites only what block $k$ adds at that position, $h_{k} - h_{k - 1}$ , with the corrupt run's, and leaves the rest of the stream clean; its effect is called $c_{k}$ . Only the last position is patched, and every other position keeps its clean state.

The original paper checked its lens causally too, but on directions: it found a strong correlation between a direction's influence on the lens and on the model [2]. The check here is coarser and about whole layers. Causal tracing on facts found its effects at the middle layers of the subject's last token (an "early site" there [15]); this page patches the last position only, so it cannot see that site. Comparisons of patching metrics have used the same last-position setup: a single layer's activation at the last token of the prompt [16], though for MLP layers alone and not whole blocks.

Go deeper: Activation Patching and Causal Tracing · Circuit Tracing and Attribution Graphs

16Checking by patching

When does the last position's stream carry the decision?

The residual patch curve $s_{ℓ}$ starts at zero, because the last token of both prompts is the same, so patching the embedding changes nothing. It ends at one, because patching the stream after the last block hands over the corrupt run's output. Where it crosses one half is the first layer at which the last position's stream alone is enough to move half the answer. Call this the causal depth.

The comparison the research desk makes is between this layer and each lens's depth. A lens cannot find at layer $ℓ$ what is not yet at the last position; but a lens may find something the model does not yet use, or miss something it already carries.

Go deeper: Activation Patching and Causal Tracing · The Residual Stream

17Checking by patching

Do the lenses and the patches pick the same blocks?

A lens implies a story about which layers matter: where the answer's log-probability rises. Write the rise across block $k$ as

g_{k} = lo g p_{k} (a) - lo g p_{k - 1} (a)

for the lens in question, and compare it with the block-patch effect $c_{k}$ across the 24 blocks using a Spearman rank correlation, $ρ (g, c)$ . A lens whose story is right should have large $g_{k}$ where $c_{k}$ is large.

Two cautions on reading $ρ$ . The log-probability under a lens that has not yet formed the answer can fall or rise by large amounts for reasons unrelated to the answer, and the rank correlation does not care about size. And a single block's patch is a small intervention that later blocks may undo, so $c_{k}$ can be near zero for a block that does matter.

Go deeper: Activation Patching and Causal Tracing · Circuit Tracing and Attribution Graphs

18Research desk

E0. Is the tuned lens more faithful at 0.5B, on this model family?

Claim

Tuned-lens predictions have substantially lower perplexity and are more representative of the final layer's distribution. [2] Shown on GPT-2, GPT-Neo, Pythia, OPT and BLOOM; Qwen2.5, with tied embeddings and grouped-query attention, was not among them.

Hypothesis

It holds: at every intermediate layer the tuned lens is closer to the final distribution than the logit lens, by a wide margin in the middle.

Design

12,700 held-out WikiText tokens (50 windows, positions 1 to 254); full-vocabulary KL(final ‖ lens) in bits at layers 1 to 23; two independent lens trainings (seeds 0 and 1, differing in minibatch order only).

Verdict rule

Supported if both seeds have the tuned lens lower at all 23 layers and the mean over layers falls by at least 50%; partly if lower at 20 or more layers in both seeds; not supported otherwise.

Reading it. By the paper's own measure the claim replicates on a model family the paper did not test, and the gap is large: the logit lens of this model is, in the middle layers, many bits from what the model finally says, and the tuned lens (trained for twenty minutes on a CPU) closes most of it. The two seeds differ in minibatch order only and agree, which says the result is not an accident of data order, not that it is robust to everything.

Go deeper: Knowledge Distillation · Probability and Information Theory

19Research desk

E1 and E2. Where does each lens find the answer, and is that where the model commits?

Claim

Logit lens: a pretty good guess forms in the middle. Tuned lens: the logit lens is "often brittle", and its readings track the model causally in the paper's own direction-level test.

E1 hypothesis

The tuned lens holds the answer for good earlier than the logit lens, in all three families.

E1 rule

Supported if the median per-prompt gain (logit depth − tuned depth) is at least 2 layers in all three prompt sets of at least two families and at least 0 everywhere else; partly if the pooled median gain is above 0 in every family; not supported otherwise.

E2 hypothesis

Because both lenses read the same stream, the tuned lens's depth lies closer to the causal depth (residual patching crossing one half) than the logit lens's does; the logit lens reads the answer later than the stream commits to it.

E2 rule

Supported if the median |depth − causal depth| is smaller for the tuned lens in all three families; partly in two; not supported otherwise.

Design

90 prompts (3 families × 3 sets × 10), each with a corrupt partner, 25 residual patches per prompt; lens seeds 0 and 1; the verdict is the weaker of the two seeds.

Reading the two results together. For facts and copies the layer at which patching first moves half the answer (the causal depth) and the layer at which either lens holds the answer for good are within about two layers of each other, and the tuned lens gains nothing. The simplest account is that the limit is the model's, not the lens's: at the last position the answer is not in the stream before then, so no reading of the stream can find it. This is consistent with the smoke test on a prompt outside the study, where a single late block carried the answer into the last position, and with the exploratory result below that one block, 22, moves most of the margin for facts. For sums the picture is different: a median causal depth of 17, but both lenses lag it by four to five layers, so here the stream carries what fixes the digit in a form a lens trained on encyclopaedia text does not read. Whether a better-trained translator, or one trained on arithmetic, would close that gap is untested.

Go deeper: Activation Patching and Causal Tracing · Induction Heads and In-Context Learning

20Research desk

E3. Do the two lenses disagree about which blocks matter, and who does patching side with?

Claim

The logit-lens post says that later layers refine earlier guesses in light of one another; the canon asks whether the two lenses disagree about which layers do the work.

Hypothesis

The tuned lens's per-block gain profile agrees better with the block-patching profile than the logit lens's does.

Design

For each of 90 prompts, Spearman ρ between g_k and c_k over the 24 blocks, for each lens; family means; also the overlap between the top three blocks by gain and by patch.

Verdict rule

Supported if the mean ρ is higher for the tuned lens by at least 0.10 in two or more families, with the same sign in all three prompt sets of each of those families; partly if higher in two or more families; not supported otherwise.

Reading it. Neither lens's layer-to-layer gain is a good map of the blocks that matter: the rank correlations with block patching are modest for both and do not separate them. That is not a failure of the lens to read the final distribution (E0 shows it does), but a reminder that the rise of an answer's probability across a layer is a different quantity from that layer's causal contribution. A single block's patch can also be undone by later blocks, so the patching profile is itself a lower bound on how much a block matters.

Go deeper: Circuit Tracing and Attribution Graphs · Activation Patching and Causal Tracing

21Research desk

What else was in the numbers, and what is left?

Labelled exploratory because they were not pre-registered, so treat them as leads.

Open questions

Train longer on more. With the 65 million tokens of the paper's recipe (or enough Muon steps to reach it) do the tuned-lens depths move earlier? The lenses here are small-data lenses.
The base model. This page used the instruct model. The paper found lenses transfer to fine-tuned versions of a base model; whether Qwen's base-model lens does here, and whether depth differs, is untested.
Whole-model patching. Patch the answer-position stream together with the subject-token streams to separate "not yet at the last position" from "not yet anywhere".
Prompts the model gets wrong. Does either lens show the correct answer forming and then being overwritten, as the paper reports for false demonstrations at a critical layer [17]?
The same on a larger model. Is the gap between lens depth and causal depth a property of 24 layers?

22Method

What would be wrong to conclude?

One model of 0.5 billion parameters. Nothing here is a forecast for larger ones. The finding that a lens reads the answer late is about this stack of 24 blocks.
Small-data lenses. The translators saw 91,800 distinct tokens for about 21 CPU minutes. A longer training could move them; the paper itself reports that SGD-trained lenses were undertrained.
Reduced vocabulary in training. The loss used 8,192 tokens. Evaluation did not.
Easy prompts. Every study prompt is one the model answers correctly.
The last position only. Patching cannot see information that later layers still fetch from earlier positions.
Lenses trained on encyclopaedia text, read on templates. Calibration on prompts of this kind was not checked.
Reading, not meaning. A lens maps a vector to a distribution over tokens. It does not say what the model represents.

23Method

How was it built, and how can it be rebuilt?

The model is Qwen2.5-0.5B-Instruct at revision 7ae557604adf, float32, on four CPU cores, with PyTorch 2.14. No GPU was available, and Ollama could not be installed (its release archive was not reachable from the sandbox). The independent truths for the page are the model itself, re-run from scratch by a different route and compared with the recording that test.mjs checks the in-page lenses against, a numpy and SciPy recomputation of every derived number, and a second runtime on the model's output. probe/ holds the recording scripts, train/ the lens training, research/ the pre-registration, the runners and the summariser; a lab note in research/README.md records every deviation, including those made after seeing a result.

parameters are counted from the loaded weights; every quotation in the ledger was checked against the fetched source text (the arXiv LaTeX source, the LessWrong post as served, the model's config file), and nothing is cited that was not read.

Cite this page as the citation block below. It reports a replication on one small model and three new tests, all pre-registered; it is not a result about language models in general.

References

nostalgebraist (2020). interpreting GPT: the logit lens. LessWrong, 30 August 2020 (with later edits) Anchor post for the logit lens. Quotes are from the post as served on 2026-10-19.
Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S. and Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112 (current version at retrieval, 2026-10-19; includes the later Muon note) Anchor paper for the tuned lens. Quotes are from the arXiv LaTeX source.
Yang, A., Yang, B., Zhang, B. et al. (Qwen Team) (2024). Qwen2.5 Technical Report. arXiv:2412.15115 The model on this page, Qwen2.5-0.5B-Instruct. Quotes are from the LaTeX source.
Qwen Team (2024). config.json of Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576. Hugging Face model repository Layer count, widths, vocabulary and norm epsilon of the exact weights studied.
Elhage, N., Nanda, N., Olsson, C. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread The residual-stream view of read and write.
Sun, M., Chen, X., Kolter, J. Z. and Liu, Z. (2024). Massive Activations in Large Language Models. arXiv:2402.17762 The outlier residual dimensions visible in the residual-stream block.
Xiao, G., Tian, Y., Chen, B., Han, S. and Lewis, M. (2023). Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453 Why the first token of a window is excluded from lens training and evaluation here.
Geva, M., Caciularu, A., Wang, K. R. and Goldberg, Y. (2022). Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. arXiv:2203.14680 Reads each feed-forward update as an additive change to a distribution over the vocabulary.
Jastrzębski, S., Arpit, D., Ballas, N., Verma, V., Che, T. and Bengio, Y. (2017). Residual Connections Encourage Iterative Inference. arXiv:1710.04773 The iterative-inference view the tuned lens builds on.
Lad, V., Gurnee, W. and Tegmark, M. (2024). The Remarkable Robustness of LLMs: Stages of Inference?. arXiv:2406.19384 Layer deletion and swapping across model families.
Yom Din, A., Karidi, T., Choshen, L. and Geva, M. (2023). Jump to Conclusions: Short-Cutting Transformers With Linear Transformations. arXiv:2303.09435 A concurrent linear-map alternative to the logit lens.
Merity, S., Xiong, C., Bradbury, J. and Socher, R. (2016). Pointer Sentinel Mixture Models. arXiv:1609.07843 Introduces the WikiText corpus used to train and evaluate the lenses here.
Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L. and Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks. Blog post The optimiser the tuned lens paper recommends and this page uses for the translators.
Heimersheim, S. and Nanda, N. (2024). How to use and interpret activation patching. arXiv:2404.15255 Noising versus denoising and the choice of metric.
Meng, K., Bau, D., Andonian, A. and Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022; arXiv:2202.05262 Causal tracing: the corrupt-and-restore design the patching here borrows.
Zhang, F. and Nanda, N. (2023). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. arXiv:2309.16042 Metrics and corruption methods for patching.
Halawi, D., Denain, J.-S. and Steinhardt, J. (2023). Overthinking the Truth: Understanding how Language Models Process False Demonstrations. arXiv:2307.09476 Uses the logit lens to find a critical layer; the tuned lens paper extends it.

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
unembed intermediate layers	the logit lens	[1] Overview If one applies the same function to the activations of intermediate GPT layers, the resulting distributions make intuitive sense.
L affine maps	translators, one per layer	[2] §1 We train L affine transformations, one for each layer of the network, with a distillation loss: transform the hidden state so that its image under the unembedding matches the final layer logits as closely as possible.
lower perplexity, closer to final	the paper's headline claim	[2] §1 We find that tuned lens predictions have substantially lower perplexity than logit lens predictions, and are more representative of the final layer distribution.
"often brittle"	the paper on the logit lens	[2] Abstract Our method, the tuned lens, is a refinement of the earlier "logit lens" technique, which yielded useful insights but is often brittle.
24 layers, 14 / 2 heads, tied	layers, query and KV heads, tied embeddings	[3] Table 1 0.5B & 24 & 14 / 2 & Yes & 32K / 8K & Apache 2.0
24	blocks	[4] config.json "num_hidden_layers": 24,
14	query heads	[4] config.json "num_attention_heads": 14,
2	key-value heads	[4] config.json "num_key_value_heads": 2,
GQA, SwiGLU, RoPE, RMSNorm pre-norm	architecture components	[3] §2 The architecture incorporates several key components: Grouped Query Attention (GQA, ) for efficient KV cache utilization, SwiGLU activation function for non-linear activation, Rotary Positional Embeddings (RoPE, ) for encoding position information, QKV bias in the attention mechanism and RMSNorm with pre-normalization to ensure stable training.
896	residual width	[4] config.json "hidden_size": 896,
4,864	MLP width	[4] config.json "intermediate_size": 4864,
151,936	embedding rows	[4] config.json "vocab_size": 151936
true	tied unembedding	[4] config.json "tie_word_embeddings": true,
151,643	regular tokens	[3] §2 with a vocabulary of 151,643 regular tokens
3 to 22	control tokens	[3] §2 We have expanded the set of control tokens from 3 to 22 compared to previous Qwen versions
pre-LN	designed for pre-LayerNorm	[2] §2, footnote Both the logit lens and the tuned lens are designed primarily for the pre-LN architecture, which is more unambiguously iterative.
linear read, additive write	read from and write to the residual stream	[5] Residual stream Both the attention and MLP layers each "read" their input from the residual stream (by performing a linear projection), and then "write" their result to the residual stream by adding a linear projection back in.
100,000 times larger	massive activations	[6] Abstract very few activations exhibit significantly larger values than others (e.g., 100,000 times larger)
attention sink	initial tokens act as a sink	[7] Abstract We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention.
additive update	each FFN update is an additive change to a vocabulary distribution	[8] Abstract We view the token representation as a changing distribution over the vocabulary, and the output from each FFN layer as an additive update to that distribution.
iterative refinement	residual blocks move along the negative gradient	[9] Abstract We formalize the notion of iterative refinement in Resnets by showing that residual connections naturally encourage features of residual blocks to move along the negative gradient of loss as we go from one block to the next.
only the first layer is crucial	layer deletion	[2] App. C.1 We find that only the very first layer is crucial for performance; every other layer ablation induces a nearly imperceptible increase in perplexity.
robust to dropping middle layers	middle layers can be dropped	[10] Abstract interventions to the early and final layers cause the most degradation, while the model is remarkably robust to dropping middle layers.
detokenization, feature engineering, prediction ensembling, residual sharpening	four stages	[10] Abstract (3) prediction ensembling, where hidden states are aggregated into plausible next-token predictions; and (4) residual sharpening, where irrelevant features are suppressed to finalize the output distribution.
1e-6	RMSNorm epsilon	[4] config.json "rms_norm_eps": 1e-06,
"pretty good guess"	a guess forms in the middle	[1] Overview At some point in the middle, GPT will have formed a "pretty good guess" as to the next token, and the later layers seem to be refining these guesses in light of one another.
later edit	a variant for hard models	[1] Edit 5/17/21 trying out a variant of the "decoder" used in this post, which dramatically helps with interpreting some models
inputs vanish after layer 0	inputs are not kept around	[1] Overview On the other hand, only the inputs look like the input tokens.
output space the whole way	no U-shape	[1] Why / is this surprising? Instead, the input space immediately vanishes, and we're in output space the whole way.
more than half the layers	logit lens predicts the input token	[2] §2 we find that for BLOOM and OPT 125M, the top 1 prediction of the logit lens is often the input token, rather than any plausible continuation token, in more than half the layers
some, not all	what the lens shows	[1] Overview The logit lens show us some of what is going on, not all of it.
covariances drift apart	representational drift	[2] §3 we observe a strong tendency for the covariance matrices of hidden states at different layers to drift apart as the number of layers separating them increases
4 to 5 bits	logit lens is biased	[2] §2 We find the bias of the logit lens can be quite large: around 4 to 5 bits for most layers.
LogitLens(Ah + b)	tuned lens formula	[2] §3 We have now arrived at the tuned lens formula, featuring a learned affine transformation for each layer:
identity	initialisation	[2] §3 We initialized all translators to the identity transform, and use a weight decay of 10^-3.
linear shortcut	linear maps from hidden layers to the last layer	[11] Abstract In this work, we suggest a simple method for such casting, using linear transformations.
more accurate	more accurate than inspecting layers directly	[11] Abstract our method produces more accurate predictions from hidden layers, across various model scales, architectures, and data distributions.
SGD, 250 steps	original training recipe	[2] §3 In our initial experiments, we used SGD with Nesterov momentum, with a linear learning rate decay schedule over 250 training steps.
2^18	tokens per step	[2] §3 We accumulated gradients as necessary to achieve a total batch size of 2^18 tokens per optimizer step.
under an hour	paper's training cost	[2] §6 our code can train a full set of probes in under an hour
freely available	WikiText corpus	[12] Abstract we also introduce the freely available WikiText corpus.
optimiser for 2D parameters	Muon	[13] Opening Muon is an optimizer for 2D parameters of neural network hidden layers.
"severely undertrained"	later correction	[2] §3 (Muon) Essentially, all of the tuned lenses trained and evaluated in the earlier versions of this paper were severely undertrained.
layers until top-1 stops changing	definition of prediction depth	[2] §5.3 we define the prediction depth of a prompt x to be the number of layers after which a model's top-1 prediction for x stops changing.
positive on all tasks	depth tracks how late an example is learned	[2] §5.3 We find a significant positive correlation between the iteration learned and the tuned lens prediction depth on all tasks we investigated.
last layers	copying is found late	[1] Copying a rare token As shown below (truncated to the last few tokens for visibility), the model correctly predicts "plasma" at the last position, but only figures it out in the very last layers.
necessary	noising patches	[14] §1 We observe which patch breaks the clean-prompt behaviour, i.e. patching which activations were necessary to maintain for the behaviour.
corruption rule	tracing depends on a corruption rule	[15] §2 We note that causal tracing depends on a corruption rule to create baseline input for a model that does not contain all the information needed to make a prediction.
logit difference	prefer logit difference	[14] §2 here they disagree, we personally trust logit difference (or equivalently logprob difference) the most.
Spearman 0.89	causal check of the tuned lens	[2] §4.1 In accordance with Property 1, there is a strong correlation between the causal influence of a feature on the tuned lens and its influence on the model (Spearman rho = 0.89).
mid-layer MLPs at the last subject token	causal tracing finds an early site	[15] Fig. 1 strongly causal states at an `early site' in middle layers at the last subject token is a new discover
last token, single layer	patching setup in the literature	[16] §5 All the experiments patch a single MLP layer's activation at the last token of the prompt.
no retraining needed	lenses transfer to fine-tuned models	[2] §3 We find that lenses trained on a base model transfer well to fine-tuned versions of that base model, with no additional training of the lens.
diverges at a critical layer	a critical layer from the logit lens	[17] Abstract At early layers, both demonstrations induce similar model behavior, but the behavior diverges sharply at some "critical layer", after which the accuracy given incorrect demonstrations progressively decreases.

How this was made

Recorded from the real model (Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576, float32 PyTorch on CPU): the residual stream at the answer position for nine fixed prompts, both lenses' full-vocabulary readings of every layer, and the patching curves of 90 prompts. Computed in the page: the logit lens at every layer and the tuned lens at layer 12 from the stored residual, RMSNorm weights, unembedding rows and translator, and every depth, gain and correlation the text states. test.mjs checks the in-page lenses against the recorded readings (worst log-probability difference reported in its output), a from-scratch re-run of three prompts by a different route, every derived number against a numpy and SciPy recomputation, the patching endpoints, the parameter count from the config values, and that files/research.json is what the summariser produces from the results. Quotations come from the arXiv LaTeX sources and the LessWrong post and were grep-checked against them. The research desk is pre-registered in research/README.md with a lab note of every deviation.

Built with the dissection skill. The model was run on a CPU; every recording and test is in probe/, train/ and research/.

Cite this page

@misc{sigmoid-lens-depth,
  title  = {Where the answer forms},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/lens-depth}
}