Mechanistic interpretability · original research

Why 722 becomes 719

A real 0.5B language model opened up at the moment it copies a number from its context: the heads that carry the digit, the votes that choose it, the sampler that sometimes doesn't, and a guard that checks every digit as it is written.

ModelQwen2.5-0.5B-Instruct, recorded

RuntimeOllama in Docker, PyTorch, RTX 5060

Reading23 min · 22 steps

Published2026-10-18

01Overview

Two people, one paragraph, two different numbers

A research summary says a study followed 722 adults. Two people paste it into the same small language model and ask for a short news article. One article says 722. The other says 719. Nothing in the paragraph says 719. This page opens the model up at the moment it writes that number, and tests, on the model's own weights, how a copied number goes wrong.

The hypothesis that started it is precise enough to test. After the model wrote "7", the probabilities of "2" and "1" were nearly equal; the second person's sample drew "1"; and once "71" was on the page, the model completed it with "9", because 719 is the closest number to 722 that begins with 71. That is three claims about the inside of a network: the slip is a near-tie, the completion is the nearest number, and something inside the model was supposed to carry the "2" across and did not quite. A fourth question follows from them: if a slip can be seen starting at a single token, can it be stopped there?

What you are looking at is that model at work, laid out on a table. At the bottom are the notes it was given and the article it is writing. In the middle are its 24 layers, drawn once for each digit of the number: every attention head's look back at the source digit, every head's and every MLP's vote for the next digit, and what the network would predict if it stopped after each layer. At the top is what decoding does with that: the sampler, every number the article could contain, and a guard that checks each digit against the notes. Every block holds values recorded from the real model or computed from them in this page. Hover any cell to read it.

The pictures follow the grammar of Brendan Bycroft's LLM Visualization [1]: tensors as blocks of cells, the camera walking through the computation in the order it happens.

How to use this page

Scroll the article, or use ← →. Drag the stage to orbit, right-drag to pan, scroll on it to zoom; O shows the whole machine, C returns to the step's view, R replays its animation. Pick a context in the bar above: four research notes, each clean or with a near-miss number added, and every block changes to that context's recording. The stage opens on the catalyst note with its near-miss (384, with 381 also in the notes), the context where this model's copy is least certain among the articles it wrote itself; the 722 note is one click away.

Go deeper: Mechanistic Interpretability Primer · The Mechanistic Interpretability section

02Overview

A real model, small enough to run on a laptop

The model is Qwen2.5-0.5B-Instruct, the smallest of the Qwen2.5 family: 24 layers; 14 query, 2 key-value heads; tied embedding, with grouped-query attention, SwiGLU [2]. It has parameters and was pre-trained on 18 trillion tokens. It is not a miniature built for this page. It is the same model anyone can download, which means every finding here is about a real pretrained network, and also that every finding is about a network of half a billion parameters, not a frontier model.

Two runtimes read it. Ollama serves it in Docker on a laptop GPU (an RTX 5060 with 8 GB) and produced every sampled article in the study, with the log-probability of every token it wrote and of the 20 most likely alternatives, a field its API describes as the "number of most likely tokens to return at each token position" [3]. PyTorch loads the same weights to read what Ollama cannot show: attention, every layer's residual stream, and what happens when parts of the network are switched off. The test suite checks that the two agree on every recorded step.

Four other models ran the behavioural study beside it: the same Qwen at 4-bit quantisation, Qwen2.5-1.5B (28 layers; 12 query, 2 key-value heads), Gemma 3 1B and Llama 3.2 1B. They are there to ask whether a finding is about this one network or about the job of copying a number.

03The context

To the model, 722 is three decisions

A language model reads tokens, not characters. How a tokeniser cuts numbers decides what copying one even means. Qwen's tokeniser, inherited by Qwen2.5 (Qwen's, byte-level BPE) [2], was built so that "we have split numbers into single digits." [4] Its vocabulary is about 152 thousand tokens, and not one of them is a multi-digit number.

The strip at the bottom of the stage is the paragraph the model was given, one cell per token. The digits are labelled; the source number is in gold. To write the model must emit three digit tokens in a row, each one a fresh draw from a fresh distribution. That is what makes the hypothesis on this page testable at all: there is a separate moment for the "7", the "2" and the final "2", and a slip has to begin at one of them.

Other models cut numbers differently. Llama 3's tokeniser "contains individual tokens for all numbers between 0 and 999 inclusive" [5], so for Llama 3.2 the number 722 is one token and a slip from 722 to 719 is a single wrong choice among a thousand number tokens, not a digit-by-digit drift. GPT-3.5 and GPT-4 use chunks of up to three digits too, while Llama 1 and 2 and PaLM use single digits (Singh and Strouse) [6], and the direction of the chunking matters: arithmetic is up to 20% more accurate when numbers are chunked from the right. The study runs a Llama arm to see whether the slip survives a different tokeniser.

Go deeper: Tokenisation and Arithmetic · Tokenisation and BPE

04The context

The moment of writing

The second strip is the article the model writes from those notes, up to the point where the number goes: "…". Everything on the stage above is recorded at exactly this point. The model has read the notes, begun its article, and is about to produce the first digit.

The instruction asked for a short news article for a general audience that mentions the key figures. That is a copying task hidden inside a writing task: the model is free to phrase everything else as it likes, but the number has one right answer, and it is sitting in the context forty or fifty tokens back.

Go deeper: Autoregressive Generation

05Inside the model

The first digit is the easy one

Each of the three columns of blocks is the whole network seen from one position: the position whose output chooses a digit. The left column chooses the first digit. The first digit of a number in a news article is rarely in doubt once the sentence has committed to "The study involved…", and on this page it never slips; every slip the study found began later.

Inside each column, three kinds of block share their rows, one row per layer with the last layer at the top. The left block is attention: for each of the 14 heads in each of the 24 layers, how much of that head's attention goes to the source digit in the notes. The middle blocks are votes, explained two steps on. The right block is the logit lens, explained after that. The bar at the top is the model's output: the probability of each digit, and of ending the number.

Go deeper: Next-Token Prediction · Softmax and Logits

06Inside the model

A few heads look straight at the digit to copy

The middle column chooses the second digit, the one the hypothesis is about. Its attention block shows something attention maps rarely show this cleanly: almost every head is dark, and a handful are nearly white. The eight brightest, ranked across the template numbers, are outlined.

This is the behaviour Wu and colleagues named in long-context models. Their retrieval heads are found by a copy-paste test, frequency of copy-paste [7], scored as

retrieval score (h) = \frac{∣ g _{h} \cap k ∣}{∣ k ∣},

where $k$ is the set of tokens to be copied and $g_{h}$ the tokens head $h$ attended to most at the moment each was copied: "a token-level recall rate of the most attended tokens". They count a head as a retrieval head if it scores above 0.1, and report that less than 5% of heads qualify. Here the ranking is the simpler quantity on the stage: attention on the source digit at the moment of copying, averaged over 30 numbers.

The idea is older than the name. Induction heads, in Olsson and colleagues' account, [A][B] … [A] → [B] [8]: the pattern $[A] [B] \dots [A] \to [B]$ , where a head finds an earlier place the current context occurred and copies what came next. A number in a context is the cleanest case there is, and the second half of their definition is the one the next steps test: raises the attended token's logit

Go deeper: Induction Heads · Attention Mechanism · Grouped-Query Attention

07Inside the model

Where those heads look, across the whole prompt

The two strips at the bottom now carry the copy heads' attention, averaged over the eight: the brighter a token, the more of their attention it gets from the position that chooses the second digit. In the notes, the strongest attention lands on the source digit the position is about to copy: in 23 of the 24 decisions recorded across the eight contexts (three digits each), the brightest token in the notes is that digit. In the article the attention is spread thin, with no single token standing out.

Attention is a correlation, not a mechanism. A head can attend to a token and do nothing useful with it, so the claim that these heads carry the copy is tested by removing them on the research desk below. The pattern is the reason to suspect them; the ablation is the test.

Go deeper: Attention as Soft Dictionary Lookup

08Inside the model

Who votes for the right digit, and by how much

Every head and every MLP adds a vector to the residual stream, the running sum that the last layer turns into logits. Because the final step is linear apart from one normalisation, each component's addition can be read as a vote: how far it moves the logit of one digit against another. This is direct logit attribution. For a component writing $x_{c}$ into a stream that ends at $x_{L}$ ,

Δ_{c} = (W_{U} [a] - W_{U} [b])^{⊤} \frac{γ ⊙ x _{c}}{RMS ( x _{L} )},

where $W_{U}$ is the unembedding, $γ$ the final RMSNorm's weight, $a$ the source digit and $b$ the strongest rival. Summed over the embedding, every head and every MLP, the votes equal the model's own logit difference exactly; the test suite checks that they do. The middle blocks of each column are these votes, heads on the left and MLPs in the thin column beside them; amber pushes toward the source digit, blue away.

Logit differences are the measure Zhang and Nanda recommend for this kind of localisation: logit difference [9]. A probability can hide a component that pushes the wrong way, which is why the votes are drawn in logits.

Go deeper: The Residual Stream · Circuit Tracing and Attribution Graphs

09Inside the model

When does the answer appear?

The right-hand block of each column is the logit lens. nostalgebraist's observation was that a transformer's final projection is a fixed function, so the unembedding applied to every layer [10]: decode the residual stream after every layer as if the model stopped there,

p_{ℓ} = softmax (W_{U} RMSNorm (x_{ℓ})),

and watch the prediction form. In GPT-2, early convergence Each row is one layer, the embedding at the bottom and the output at the top; each column a digit. For most of the network the digit is not readable at all; it appears late, after the layers where the copy heads sit, and then fast.

Go deeper: Unembedding and the Logit Lens · Superposition

10Inside the model

Three distributions, and each one is lopsided

The bars at the top of the three columns are the model's output at each decision: ten digits and the option of ending the number. On the copy path the source digit wins every one of them, usually with all but a fraction of a percent, in every context but one: after the catalyst note's fixed lead-in the model does not write 384 at all. A copied number goes wrong only when decoding draws one of those fractions, or when the model's own favourite is already wrong.

That gap, measured in nats (natural-log units), is the quantity the first research question is about. A gap of 1 nat means the wrong token has $1/ e \approx 37%$ of the right one's probability: a near-tie. A gap of 10 nats means a ratio of about 22,000 to 1.

Go deeper: Log-Probabilities · Calibration of Language Models

11Decoding

Temperature decides how often a fraction gets drawn

A model outputs a distribution; something else picks a token. With temperature $T$ , the logits $z_{i}$ are divided before the softmax:

p_{i} (T) = \frac{exp ( z _{i} / T )}{\sum _{j} exp ( z _{j} / T )} .

Below 1 the distribution sharpens toward the favourite; above 1 it flattens, and the long tail of unlikely tokens gets a real chance. Ollama's documented default is 0.8 [3]. Its default repetition penalty is 1.0, which turns it off, so nothing else reshapes the distribution here; the study passes every setting explicitly anyway.

The sampler station above the first column takes the model's distribution after the first digit through each stage of decoding. The figure below drives it: move the temperature and watch the second and third bars change, and with them the probability, computed exactly, that this article writes the right number.

Go deeper: Temperature Sampling · Greedy Decoding

12Decoding

Top-k and top-p cut off the tail

Holtzman and colleagues argued that the danger in sampling is not the favourite but the crowd behind it: tens of thousands of tokens [11]. Their fix, truncating the unreliable tail, is to sample only from the nucleus: smallest set with mass ≥ p The older alternative keeps a fixed number of candidates: the k most likely tokens

V^{(p)} = the smallest set with x \in V^{(p)} \sum P (x) \geq p .

Ollama applies both by default, top-k 40 and top-p 0.9, in a fixed order the page reproduces: keep the 40 most likely tokens, apply the temperature, renormalise, then keep the smallest set holding 90% of the mass. The order matters; the pilot that preceded the study checked it against Ollama's own draws, and the test suite checks it on every recorded node.

For a copied digit, truncation is mostly protective: at a step where the right digit holds 99.9%, top-p of 0.9 removes every other candidate, and a slip becomes impossible. It stops protecting at a near-tie, because a rival with a third of the mass is inside the nucleus. That is why the first research question asks what the gaps at real slips look like.

Go deeper: Nucleus Sampling · Top-k Sampling · Min-p and Typical Sampling

13Decoding

Every number the article could contain

Because each digit is a separate token, the set of numbers the model can write here is a tree: one distribution at the start, one after each likely first digit, one after each likely pair. The block of rows above the model is that tree, recorded from the model at every prefix with at least a 0.2% chance, shown under the current sampler. Multiplying down its branches gives the exact probability of every complete number, which is the bar row above it.

This is the page's truth model. It runs the sampler and the guard in your browser on the recorded tree, and the test suite holds it to account against real decoding: for three contexts and two sampler settings, 1,000 Ollama samples continuing the same article must land within four standard errors of the page's prediction for every outcome above 1%.

At Ollama's default settings, every one of the model's own greedy articles makes a slip at this point all but impossible: the right digit's lead survives temperature 0.8 and top-p 0.9 removes every rival. Raise the temperature to 1 with top-p off and the slips appear. The slips the study found in sampled articles came from wordings the model had sampled earlier in the same article, which put it in less certain states than its greedy one; this tree is one prefix, not all of them.

Go deeper: Sampling and Decoding · Decoding Determinism

14Decoding

Person A and person B were never using the same draw

The two people in the opening used the same model and the same paragraph, and got different articles, because the sampler drew different random numbers. Ollama's documentation lists the seed's default as 0, but its code sets it to −1, a fresh seed per request, so two unseeded requests differ by design. The figure above draws two such readers from the recorded tree; "Find a slip" searches seeds until one of them writes a different number.

Whatever the model writes goes back into the article and becomes context for every later token, the gold loop on the stage. A slipped digit is not corrected later by the same mechanism that produced it; the next digit is chosen given the slip. What the model does with a number that has already gone wrong is the second research question.

Go deeper: Decoding Determinism

15Research desk

Four claims, written down before any run

The research desk turns the opening hypothesis into four tests, each with its design and a numeric verdict rule written in research/README.md before any study run. Two pilot runs on prompts outside the study checked only the tooling: that Ollama's log-probabilities are the model's raw ones (identical at temperatures 0, 0.8 and 1.5), that its unset sampler behaves like top-k 40 with top-p 0.9, and that Qwen splits 4815 into four tokens.

The slip is a near-tie: the wrong digit is drawn where the right digit's lead is small, not from the far tail.

After a slip, the model completes toward the closest number (71 → 719), not the source's next digit (71 → 712).

A few heads carry the copy; mean-ablating them makes the source digit unlikely, while random heads do not.

A guard that checks each digit against the context's numbers as it is generated stops slips without breaking numbers the model must derive.

Materials: four research notes (sleep, catalyst, ecology, trial), each with one target (722, 384, 561, 947) and two or three other numbers, each in a clean version and a version with a near-distractor sentence (719, 381, 568, 941). Behaviour is measured on 960 articles for each Qwen 0.5B arm and 640 for each comparison model; internals on 60 template numbers split into a half that ranks and a half that tests. Lab notes record every mistake that shaped the analysis, including two bugs found and fixed before any verdict was applied.

16Research desk

E1. How often does the number survive?

Question

Across models and samplers, how often is the target copied exactly, and where do slips begin?

Design

Six arms, eight contexts, seed blocks of 40: three blocks (960 articles) for the Qwen 0.5B arms and two (640) for the comparison models, up to 160 tokens each, with the log-probabilities of every step. Arms differ only if their 95% bootstrap intervals do not overlap.

So where did 719 come from?

Go deeper: RAG Evaluation and Groundedness

17Research desk

H1. Is the slip a near-tie?

Claim

The owner's: after "7", the probabilities of "2" and "1" were almost equal.

Measure

For every step of every mention, the probability under the arm's own sampler of leaving all the context's numbers, summed, and split by the gap at that step. Supported if at least 60% of the slip mass comes from steps where the best wrong token has at least 1/e of the best right token's probability.

A second part of the claim is the shape of the near-tie: was the rival of "2" its neighbour "1"? Across copy steps on the template numbers:

Go deeper: Epistemic and Aleatoric Uncertainty · Entropy-Adaptive Sampling

18Research desk

H2. After "71": the closest number, or the next digit?

Claim

Once "71" is written, the model completes the closest number to 722, so 719.

Rival

Levy and Geva report that across digits, not around the value [5]. If digits are copied one by one, the model should write the source's third digit: 712.

Design

Teacher forcing on 60 template numbers abc: force a(b−1) and a(b+1) and read the next digit. Closest predicts 9 after a down-slip and 0 after an up-slip; copy predicts c. Replicated on Qwen 1.5B.

The right-hand column of the stage now shows the third decision after a forced slip.

Levy and Geva's own example is the clearest statement of the alternative: 633 or 823 for 833, for a correct answer of 833. They find errors close in string edit distance but not in value, about 80% of them in a single digit, and a representation in which a circle per digit, base 10, read out by probes with 0.91 and 0.92 accuracy. A model that represents numbers as digits and copies them as digits has no reason to look for the nearest value once a digit has gone wrong.

19Research desk

H3. Are the copy heads necessary?

Claim

Wu et al.: hallucination, while no effect on retrieval [7].

Design

Rank the 336 heads by attention on the source digit (template half A). On half B and the study contexts, mean-ablate the top 4, 8 and 16 (each head's output replaced by its mean over the positions of the same forward pass), and 5 random sets of the same size as controls. Supported at k = 8 if the source digit's probability falls by at least 0.30 while random sets lower it by less than 0.10.

Wu et al. also report that during hallucinated generation, their retrieval heads attended to the first token: the attention sink. In their models, a failed retrieval did not look at the wrong token; it looked at no content token at all. Their summary claim: activation explains factuality Whether the same holds for the slips on this page is untested here; the recordings cover the copy and the forced-slip paths, not sampled slips.

Go deeper: Activation Patching and Causal Tracing · Attention Sinks

20Research desk

H4. Can a per-token guard stop it?

If a slip starts at one identifiable token, a decoder can check every digit as it is produced. Both guards act only inside a number whose digits so far begin some number in the context, so the first digit is never touched, and both replace the sampled token with one that stays on a context number:

Strict: if the sampled token would leave every context number (a wrong digit, an extra digit, or ending early), emit the most probable token that stays, if it has at least 0.1% probability.

Argmax-revert: only if the sampled token leaves and the greedy token would have stayed, emit the greedy token. G2 never changes what greedy decoding writes.

Cost

Four tasks whose right answer is not in the context (a percentage, a difference of years, a total, a ratio), to count what the guards break.

This is constrained decoding, the technique behind structured output, where implementations set the probabilities of invalid tokens to zero at each step [12], applied to a constraint nobody declared: stay on the numbers you were given. A softer relative is context-aware decoding, with and without the context [13]; it reported a 14.3% factuality gain in summarisation, with a dial $α$ where 0 is ordinary decoding

Go deeper: Structured Generation and Constrained Decoding · Logit Bias and Masking · Contrastive Decoding

21Research desk

Try the guard on this context

The sampler figure above has a guard control; the stage's sampler station, tree and outcome bars follow it. Set the temperature high, switch from no guard to G1, and the slips in the outcome bars collapse into the target. Switch to G2 and only the slips that came from an unlucky draw disappear; a slip that is the model's own favourite survives it. On a context with a near-distractor, both guards accept the distractor: it is a number in the context, and the guard cannot tell which one belongs where.

22Research desk

What this piece found, and what to test next

Next experiments, each runnable with the scripts in this piece's research/ directory:

Trigger the guard from the copy heads. Engage the check only when the top copy heads put most of their attention on a context number, not whenever a prefix matches. If that separates copying from deriving, it keeps G1's fidelity without its cost on 72.2%.
Tokenisers that chunk. Repeat H2 on Llama 3.2, where 722 is one token: is a slip there a near neighbour in value, as the closest-number hypothesis predicts, now that the digit-by-digit mechanism is gone? Levy and Geva's 0 to 999 tokens make the comparison exact.
Distance and distractors. Vary how far back the number sits and how many similar numbers surround it, and test whether slip rate tracks the copy heads' attention, as Wu et al.'s account of retrieval would predict.
Patch, not only ablate. Patch the copy heads' outputs from a context with a different number and check that the written number moves digit by digit, a stronger causal test than ablation, done with the logit-difference metric Zhang and Nanda recommend.
Quantisation and scale. E1's 4-bit and 1.5B arms give the behavioural answer; the internal one is whether quantisation narrows the copy heads' margin at the second digit.

Every result above is from one 0.5-billion-parameter model, on eight synthetic research notes, with three-digit numbers. It is evidence about how this network copies a number from its context, not a forecast for larger ones. The model, the recordings, the analysis and every experiment are in the piece's directory; the scripts reproduce the study end to end on any machine with Docker and a modest GPU.

References

Bycroft, B. (2023). LLM Visualization. bbycroft.net The visual grammar of tensors as blocks of cells.
Yang, A., Yang, B., Zhang, B. et al. (Qwen Team) (2024). Qwen2.5 Technical Report. arXiv:2412.15115v2 The model on this page, Qwen2.5-0.5B-Instruct.
Ollama contributors (2026). Ollama: Modelfile parameters, API reference and default options. github.com/ollama/ollama at commit 36a6e99 (docs/modelfile.mdx, docs/openapi.yaml, api/types.go); server 0.40.6 The runtime this page's samples came from.
Bai, J., Bai, S., Chu, Y. et al. (Qwen Team) (2023). Qwen Technical Report. arXiv:2309.16609v1 The tokenizer Qwen2.5 inherits.
Levy, A. A. and Geva, M. (2024). Language Models Encode Numbers Using Digit Representations in Base 10. arXiv:2410.11781v2 (preprint) Anchor paper for how numbers are represented. Quotes are from the v2 LaTeX source.
Singh, A. K. and Strouse, D. J. (2024). Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs. arXiv:2402.14903v1
Wu, W., Wang, Y., Xiao, G., Peng, H. and Fu, Y. (2024). Retrieval Head Mechanistically Explains Long-Context Factuality. arXiv:2404.15574v1 (preprint) Anchor paper for the copy mechanism. Quotes are from the v1 LaTeX source.
Olsson, C., Elhage, N., Nanda, N. et al. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread; arXiv:2209.11895v1
Zhang, F. and Nanda, N. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024; arXiv:2309.16042v2
nostalgebraist (2020). interpreting GPT: the logit lens. LessWrong, 31 August 2020
Holtzman, A., Buys, J., Du, L., Forbes, M. and Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020; arXiv:1904.09751v2
Willard, B. T. and Louf, R. (2023). Efficient Guided Generation for Large Language Models. arXiv:2307.09702v4
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L. and Yih, S. W. (2023). Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. arXiv:2305.14739v1

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
24 layers; 14 query, 2 key-value heads; tied embedding	Qwen2.5-0.5B layers, heads	[2] Table 1 0.5B & 24 & 14 / 2 & Yes & 32K / 8K & Apache 2.0
grouped-query attention, SwiGLU	attention and MLP type	[2] §2 Grouped Query Attention (GQA) for efficient KV cache utilization, SwiGLU activation function for non-linear activation
18 trillion tokens	pre-training data	[2] Abstract we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens.
most likely tokens per position	Ollama top_logprobs	[3] docs/openapi.yaml, GenerateRequest and ChatRequest Number of most likely tokens to return at each token position when logprobs are enabled
28 layers; 12 query, 2 key-value heads	Qwen2.5-1.5B layers, heads	[2] Table 1 1.5B & 28 & 12 / 2 & Yes & 32K / 8K & Apache 2.0
Qwen's, byte-level BPE	Qwen2.5's tokenizer	[2] §2 For tokenization, we utilize Qwen's tokenizer, which implements byte-level byte-pair encoding
one token per digit	Qwen's tokenizer and numbers	[4] §2.2 we have split numbers into single digits.
about 152K	vocabulary size	[4] §2.2 The final vocabulary size is approximately 152K.
one token for each of 0 to 999	Llama 3's number tokens	[5] §3.1 Llama's tokenizer contains individual tokens for all numbers between 0 and 999 inclusive
single digits, or chunks of up to three	how models tokenise numbers	[6] Abstract with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and GPT-4 have separate tokens for each 1-, 2-, and 3-digit numbers.
up to 20% more accurate	right-to-left chunks help	[6] §1 We find that model accuracy is up to 20% higher when using R2L tokenization
frequency of copy-paste	retrieval score	[7] §2 We define the retrieval score as the frequency of a head's copy-paste operations.
token-level recall	retrieval score, in words	[7] §2 Intuitively, retrieval score represents a token-level recall rate of the most attended tokens by an attention head.
0.1	retrieval-head threshold	[7] §2 we set the threshold as 0.1, meaning that as long as the head performs copy-paste in 10% of the times, we consider it a retrieval head.
less than 5%	share of heads that are retrieval heads (abstract)	[7] Abstract only a small portion (less than 5%) of the attention heads are retrieval.
[A][B] … [A] → [B]	induction heads	[8] Introduction induction heads “complete the pattern” by copying and completing sequences that have occurred before.
raises the attended token's logit	the copying property	[8] Key concepts The head's output increases the logit corresponding to the attended-to token.
3% to 6%	share above the 0.1 threshold (§3.1)	[7] §3.1 Only about 3% to 6% of the attention heads have a retrieval score larger than 0.1
logit difference	recommended metric	[9] §6 We find logit difference a convincing metric for localization in language models.
the unembedding applied to every layer	the logit lens	[10] Overview If one applies the same function to the activations of intermediate GPT layers, the resulting distributions make intuitive sense.
early convergence	what the lens shows	[10] Overview These distributions gradually converge to the final distribution over the layers of the network, often getting close to that distribution long before the end.
partial	the lens's limit	[10] Overview The logit lens show us some of what is going on, not all of it.
0.8	Ollama default temperature	[3] docs/modelfile.mdx, parameter table The temperature of the model. Increasing the temperature will make the model answer more creatively. (Default: 0.8)
1.0, disabled	Ollama default repeat penalty	[3] docs/modelfile.mdx, parameter table Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.0, disabled)
tens of thousands of tokens	size of the tail	[11] §1 This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.
truncate it	the unreliable tail	[11] Abstract Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution
smallest set with mass ≥ p	nucleus (top-p) sampling	[11] §3.1 In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold p.
the k most likely tokens	top-k sampling	[11] §3.2 At each time step, the top k possible next tokens are sampled from according to their relative probabilities.
40	Ollama default top_k	[3] docs/modelfile.mdx, parameter table Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative. (Default: 40)
0.9	Ollama default top_p	[3] docs/modelfile.mdx, parameter table Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)
0 in the docs, -1 in the code	Ollama default seed	[3] docs/modelfile.mdx; api/types.go DefaultOptions Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. (Default: 0)
across digits, not around the value	how numerical errors are distributed	[5] Abstract LLM errors on numerical tasks are often distributed across the digits of the answer rather than normally around its numeric value.
633 or 823 for 833	an example of digit-space errors	[5] §1 LLMs are more likely to generate errors with high string-similarity to the correct answer, such as "633" or "823", than natural errors like "831" or "834"
close in edit distance, not in value	the error pattern	[5] §2 We find that errors are distributed in a digit-wise manner, where an incorrect prediction is close to the correct answer in string edit distance but not in value space.
about 80%	errors in one digit	[5] §2, Task 1 we observe that about 80% of the errors are in a single output digit, which is often not the units digit.
a circle per digit, base 10	the representation	[5] Abstract we show that LLMs internally represent numbers with individual circular representations per-digit in base 10.
0.91 (Llama 3 8B), 0.92 (Mistral 7B)	probe accuracy, base 10	[5] §3.2 The highest accuracy of 0.91 for Llama 3 8B and 0.92 for Mistral 7B is achieved when reconstructing the numbers in base 10.
6%, 3%, 8% (units, tens, hundreds)	errors by digit in the comparison task	[5] §2, Task 2 Table 1 shows that errors are distributed approximately equally between the digits.
exact target 15% of the time (hundreds)	editing one digit's representation	[5] §3.3 For the hundreds digit, the exact intended result was achieved 15% of the time, while 47% of the results were 'close' to the intended number
hallucination	pruning retrieval heads	[7] Abstract completely pruning retrieval heads leads to failure in retrieving relevant information and results in hallucination
no effect on retrieval	pruning random heads	[7] §1 masking out random non-retrieval heads, the model still find full needle;
avoid probability	patching metric	[9] §6 We generally recommend avoiding using probability as the metric, given that it may fail to detect negative model components.
a general concern	interventions off-distribution	[9] §6 More broadly, this presents a challenge to any intervention techniques that introduce OOD inputs to the model or its internal layers, including ablations.
the first token	where retrieval heads look while hallucinating	[7] §4.1 During hallucinated generation, retrieval heads predominantly attended to the initial token of the input
activation explains factuality	the headline claim	[7] §1 Activation of retrieval heads explains whether the output is factual or hallucinated.
invalid tokens set to probability zero	guided generation	[12] §1 in order to determine which tokens are valid--according to the constraints and previously sampled tokens--and setting the probabilities of invalid tokens to zero.
with and without the context	what CAD contrasts	[13] Abstract a contrastive output distribution that amplifies the difference between the output probabilities when a model is used with and without context.
14.3% (LLaMA, summarisation)	CAD factuality gain	[13] Abstract including OPT, GPT, LLaMA and FLAN-T5 for summarization tasks (e.g., 14.3% gain for LLaMA in factuality metrics).
0 is ordinary decoding	context-aware decoding's α	[13] §2.2 Larger α means more weight on our adjustment (α=0 reduces to regular decoding).

How this was made

Internals are recorded from the real model (Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576, float32 PyTorch on an RTX 5060) for eight study contexts; next-token distributions at every recorded prefix come from both PyTorch and Ollama 0.40.6 serving the fp16 weights, and test.mjs checks they agree (same top-1 token, digit log-probabilities within 0.15 nats). The sampler, guards and outcome probabilities are computed in the page and checked against 1,000 real Ollama samples per setting from the same prefix (every outcome above 1% within four standard errors, total variation under 0.05). Direct logit attribution is checked to sum to the model's own logit differences, the logit lens's last layer to equal the output, and the parameter count to match the config. Literature numbers are quoted from the arXiv LaTeX sources, the LessWrong post and the Ollama repository at a pinned commit. The research desk was pre-registered in research/README.md; every experiment is reproducible from research/ and probe/.

Built with the dissection skill for the Mechanistic Interpretability section. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization. Recordings, experiments and every script: anatomy/mech-interp/2026-10-18-number-fidelity/ in the site's repository.

Cite this page

@misc{sigmoid-number-fidelity,
  title  = {Why 722 becomes 719},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/number-fidelity}
}