Meng et al. (2022) located factual recall in the MLPs at the last subject token of a prompt, by corrupting the subject and restoring one internal state at a time, and then edited a fact by changing one matrix. Hase et al. (2023) found that where a fact traces is unrelated to where it edits best. This piece runs both on Qwen2.5-0.5B-Instruct, a real 24-block model, with 72 CounterFact facts in three sets, two noise draws each, traces of hidden states, five-block MLP windows and five-block attention windows at every token and layer, a test that freezes the MLPs of the subject token, and 288 ROME edits (36 facts at 8 layers) scored by Hase et al.'s three metrics. The stage lays the paper's grid out as blocks, redraws the rank-one update at true proportions, and sets the traced curve beside the edit scores. Seven hypotheses were written down before the runs. The early site at the last subject token reproduced in position (MLP over attention by 5 to 12 times, attention at the final token) but sits at layer 2 of 24, not in the pre-registered middle band; freezing the MLPs removed most of the early layers' effect and freezing attention almost none; editing worked best at layers 1 to 4; the layer explained most of the rewrite score's variance (67%) with the trace adding 0.2%, as Hase et al. found, but on the overall score the within-layer correlation was negative; and the traced layer's advantage over other layers is mostly the general advantage of early layers.
The machine
ROME's own figure as blocks on a table: tokens run down, the 24 layers run across. One prompt gives five maps (the size of each clean hidden state, how far the noised run drifts, and the three restoration traces); below them the average over 72 facts by token class, a freezing test, and the traced curve beside the edit scores. At the lower left the rank-one update (k*, C^-1 k*, W, the update, W plus the update) is drawn pooled from the real 896 x 4,864 down-projection, and at the far left one Qwen block at true proportions. Colour and height follow the kit: probability bars for restored effects, activations for norms, weights for matrices.
What is real. Recorded from the real model (Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576, float32 PyTorch on CPU): for 72 facts the probability of the true object under the clean run, the noised run (two draws) and every restoration, plus state norms, and for 288 edits the scores. Computed in the page: every indirect effect and share, the token-class averages, the E1 ratios and verdicts, regressions, Spearman correlations, the bootstrap and permutation tests, and the rank-one algebra on the shipped vectors. test.mjs checks the engine against numpy and SciPy from the raw results, the recordings against a second route (library forward with hooks, 54 cells) and a second runtime (Ollama, all 72 prompts, same top-1 token), the method's invariants, the edit scores recomputed from stored probabilities, the parameter count, and every static number in the story. Quotations come from the arXiv LaTeX sources, the released hyper-parameter files and the model card, and were grep-checked. The research desk is pre-registered in research/README.md with a lab note.
Working model · 3D
Mechanistic Interpretability · Causal methods
