Copying a figure from the context is the simplest grounded-generation task, and a wrong digit is the most checkable hallucination. This piece takes a real open model, Qwen2.5-0.5B-Instruct, served by Ollama in Docker on a laptop GPU and read with PyTorch on the same weights, and records its internals at the moment it writes a number it read in research notes: every head's attention on the source digit, every head's and MLP's direct vote for the next digit, the logit lens through all 24 layers, and the next-token distribution at every likely prefix of the number. The sampler, the probability of every number the article could contain and two grounding guards run live on those recordings, checked against real Ollama samples. A research desk tests the hypothesis that started it (after 7, the 2 and the 1 were nearly tied, and 71 was completed as the closest number, 719) with four pre-registered experiments across five models: where slip probability comes from, what the model writes after a forced slip, whether ablating a handful of copy heads breaks the copy, and what a per-token guard fixes and what it breaks.
The machine
A real network on a table. At the bottom, the notes the model read and the article it is writing, as token strips lit by where the copy heads look. In the middle, Qwen2.5-0.5B once per digit decision: a 24 × 14 block of every head's attention on the source digit, a block of every head's vote for the right digit over its rival with the 24 MLPs beside it, and the logit lens from the embedding to the output. At the top, the next-token distribution, the sampler (raw, temperature, top-k and top-p, guard), the digit tree of every likely prefix and the probability of every whole number. To the side, the grounding guard: the context's numbers as a trie.
What is real. Internals are recorded from the real model (Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576, float32 PyTorch on an RTX 5060) for eight study contexts; next-token distributions at every recorded prefix come from both PyTorch and Ollama 0.40.6 serving the fp16 weights, and test.mjs checks they agree (same top-1 token, digit log-probabilities within 0.15 nats). The sampler, guards and outcome probabilities are computed in the page and checked against 1,000 real Ollama samples per setting from the same prefix (every outcome above 1% within four standard errors, total variation under 0.05). Direct logit attribution is checked to sum to the model's own logit differences, the logit lens's last layer to equal the output, and the parameter count to match the config. Literature numbers are quoted from the arXiv LaTeX sources, the LessWrong post and the Ollama repository at a pinned commit. The research desk was pre-registered in research/README.md; every experiment is reproducible from research/ and probe/.
Simulation · 3D
Architecture · Distributed systems
