Two methods turn a language model's hidden state at layer 12 into a probability distribution over words: the logit lens, which applies the model's own final normalisation and unembedding, and the tuned lens, which first applies a translator trained for that layer. This piece takes Qwen2.5-0.5B-Instruct, a real 24-layer model, and lays out its residual stream at the answer position as 25 tiles of 896 recorded values, with both lenses' reading of every layer beside it. The logit lens is computed in the page from the recorded stream, the model's own RMSNorm weights and the unembedding rows of 1,500 candidate tokens; the tuned lens is computed live at layer 12 from the stored translator. A research desk, pre-registered before any run, tests four things on 90 prompts in three families (facts, copies, sums): whether the tuned lens is closer to the model on held-out text, at what depth each lens holds the answer for good, whether that depth matches the layer at which patching the last position's stream moves half the answer, and whether the lenses and block-by-block patching name the same blocks. The tuned lens proved far more faithful on held-out text (closer at all 23 intermediate layers, in both training seeds), but on the 90 prompts neither lens found the answer earlier than the other, and neither one's layer-by-layer gains tracked block patching: three of the four hypotheses were not supported, and the page says what the numbers do show.
The machine
A real network on a table, one position deep. At the bottom, the prompt as a strip of tokens with the answer position lit. Up the middle, the residual stream after 0, 1, ... 24 blocks as 25 tiles of 8 x 112 cells, one cell per hidden dimension, blue for negative and amber for positive; to their left, how much each block's attention and MLP wrote; to their right, the answer's probability and the top token under the logit lens and under the tuned lens at every layer, and the two patching checks. At the top, the model's own output and the unembedding. Far left, one block drawn at true proportions (1 cell = 8 dimensions), the live layer's translator pooled to 56 x 56, and the live lens station.
What is real. Recorded from the real model (Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576, float32 PyTorch on CPU): the residual stream at the answer position for nine fixed prompts, both lenses' full-vocabulary readings of every layer, and the patching curves of 90 prompts. Computed in the page: the logit lens at every layer and the tuned lens at layer 12 from the stored residual, RMSNorm weights, unembedding rows and translator, and every depth, gain and correlation the text states. test.mjs checks the in-page lenses against the recorded readings (worst log-probability difference reported in its output), a from-scratch re-run of three prompts by a different route, every derived number against a numpy and SciPy recomputation, the patching endpoints, the parameter count from the config values, and that files/research.json is what the summariser produces from the results. Quotations come from the arXiv LaTeX sources and the LessWrong post and were grep-checked against them. The research desk is pre-registered in research/README.md with a lab note of every deviation.
Working model · 3D
Mechanistic Interpretability · Hallucination & faithfulness
