Dissection · Meng et al., 2022 · Hase et al., 2023

Locating a fact

Causal tracing and rank-one editing on a real 0.5B language model: where restoring a state brings a fact back, where editing a matrix changes it, and whether those are the same place.

PapersMeng et al. 2022 (causal tracing, ROME); Hase et al. 2023

ModelQwen2.5-0.5B-Instruct, 494M parameters, real weights

Reading32 min · 26 steps

Published2026-10-21

01The question

Where in a language model is a fact kept?

Give a language model "" and it puts most of its probability on . Meng, Bau, Andonian and Belinkov asked which internal states that answer depends on, found a small set, and used it to rewrite a fact by editing one matrix [1]. Hase and colleagues then asked whether the place where a fact can be traced is the place where it is best edited, and reported that it is not [2].

The first paper's method, causal tracing, is a way of asking an intervention question of a trained network. Damage the part of the prompt that names the subject, so that the answer collapses. Then put the undamaged internal state of one token at one layer back into the damaged run and see how much of the answer returns. The paper's finding, in its own words, was that the strongly causal states appear at an early site at the last token of the subject, which is a new discovery. Its summary of where a fact is placed has three parts: MLP, middle layers, last subject token

The stage shows this on a real model, Qwen2.5-0.5B-Instruct, with 24 blocks, not on the paper's GPT-2 XL. One prompt lies across the top as five maps of tokens by layers; below it are the averages over 72 prompts, a test that freezes parts of the network, and the edit that the same paper builds on the finding, drawn at true proportions. The fact on the stage is : → , which has probability in the clean run and once the subject is noised.

This piece asks two questions of that model, written down before any run. E1: does the early site at the last subject token, carried by middle-layer MLPs, appear in a model with half the depth of GPT-2 XL, and what happens when those MLPs are frozen? E2: if the answer to E1 is yes, does editing at the layer where a fact traces work better than editing elsewhere, as the first paper's own results suggested and the second paper's did not?

Go deeper: Mechanistic Interpretability Primer · Activation Patching and Causal Tracing · Locate-then-Edit Knowledge Editing

02The question

Which model is being traced, and how does it differ from the paper's?

This is the released model, not a miniature. The model card gives 24 and 14 query, 2 key-value [3], and the architecture as transformers with RoPE, SwiGLU, RMSNorm, attention QKV bias and tied word embeddings. Its sizes come from config.json at revision 7ae5576 [4]: a total of 0.49B parameters (0.36B without the embedding), a residual width of 896, an MLP width of 4,864, and 151,936 embedding rows that are also the output layer (tied). The technical report says the series was pre-trained on 18 trillion tokens [5]. The paper's main model is GPT-2 XL, 1.5B parameters with 48 layers; its public configuration gives a residual width of 1,600 and 25 heads [6].

Three differences matter for what follows. Qwen's MLP is gated: it computes $W_{down} (silu (W_{gate} x) ⊙ W_{up} x)$ , so the vector that the paper calls the key, the input of the second matrix, is the product of two projections. Its attention shares each key-value head between seven query heads. And it has half the layers, so the "middle layers" of the hypothesis are about 12 blocks wide at most, and the 10-layer windows the paper restored at once would span nearly half the network.

The block drawn at the far left of the stage is to scale and carries no data: one cell is 32 hidden dimensions, so the residual stream is 28 cells wide, the 14 query heads are 14 groups of 2 cells, and the MLP's hidden layer is 152 cells, 5.4 times the stream.

Go deeper: Anatomy of a Transformer Block · Feed-forward and SwiGLU · Multi-Query and Grouped-Query Attention · Counting Transformer Parameters

03Setup

What exactly is a "fact" here?

The paper writes a fact as a tuple $t = (s, r, o)$ : a subject, a relation and an object. A prompt $p$ describes $(s, r)$ and the model's prediction of $o$ is read after it. The data are CounterFact, 21,919 records each holding a prompt template, its subject, the true object and a false one to edit in [1]. The stage's fact is : , whose subject spans of its tokens. The true object is and the false object used for editing is . All probabilities on this page are for the first token of the object, read at the last position.

The study set is chosen by a rule fixed in advance, not by looking: the 21,919 records were shuffled with a seed, and a record entered if the model's top token after the prompt was the true object's first token with probability at least 0.2 (a known fact), and the total effect defined below was at least 0.10. Dealt in turn to three sets of 24, A, B and C, these are 72 facts, no fact in two sets. Of 743 records examined, 670 were not known to the model by this test and one known fact failed the effect threshold. (Hase and colleagues kept the facts GPT-J completes, a completion accuracy of 32.6% under their scheme; a 0.5B model passes the top-token test for about one record in ten.) The three sets are the replicates of the headline question. Change the fact on the stage with the box above it (A1 to C24).

Go deeper: Log-probabilities and Token Probabilities · Byte-Pair Encoding

04Setup

What is being restored, and where does it live?

The paper draws the network as a grid: one hidden state $h_{i}^{(l)}$ for each token $i$ and layer $l$ , where each layer adds the output of attention and of the MLP to the previous state [1]:

h_{i}^{(l)} = h_{i}^{(l - 1)} + a_{i}^{(l)} + m_{i}^{(l)}, m_{i}^{(l)} = W_{proj}^{(l)} σ (W_{fc}^{(l)} γ (a_{i}^{(l)} + h_{i}^{(l - 1)}))

Here $a_{i}^{(l)}$ is attention's write, which reads the earlier tokens' states, $m_{i}^{(l)}$ is the MLP's write, $γ$ is the normalisation and $σ$ the nonlinearity. In Qwen the same equations hold with RMSNorm for $γ$ and the gated product for $σ$ . On the stage the leftmost map is this grid for the clean run, with each cell drawn as the size of the 896-number state, so the blocks are the real thing and the amber height is its norm. Layer $l$ here means the output of block $l$ (counting from 0), as in the paper's figures.

In every one of the 72 prompts that is between 77 and 88 times the median of the other tokens, whose norms at layer 12 differ from one another by a median factor of 1.2. The huge first column is the first token, where the stream is far larger than anywhere else, the pattern usually called an attention sink; the norms below it are small and even, which is why noise at the scale used later is large next to them.

Go deeper: The Residual Stream · Residual Connections · Attention Sinks · Normalisation

05Setup

How do you take the subject away from a model?

The corrupted run adds noise to the embeddings of the subject tokens before the network runs: $h_{i}^{(0)} := h_{i}^{(0)} + ϵ$ with $ϵ \sim N (0, ν)$ for each subject token $i$ . The paper sets $ν$ to 3 times larger than the empirical standard deviation of embeddings [1]. Here the standard deviation of the embedding table's entries over the real vocabulary is , so $ν =$ (Hase and colleagues, following the paper, add Gaussian noise with σ=0.094 on GPT-J), with two independent draws per fact (the paper averaged ten). The second map on the stage is the noised run's state minus the clean state, by token and layer. Tokens before the subject do not move at all, because attention only looks backwards. At layer 2 the drift is larger than the clean state's own norm for most subject tokens (213 of the 320 in the 72 prompts) and for none of the 239 tokens after the subject, which move less.

The corruption does what it is for: is the probability of under noise for this fact, against clean, a total effect of . The figure shows all 72.

Across the 72 facts the noise removes most of what the model assigned to the true answer, as in the paper, where it took the average from 27.0% to 8.47%. The facts here are selected to have total effect of at least 0.10, which only one known fact failed; at this scale the filter hardly binds.

Go deeper: Structural Causal Models and Counterfactuals · Probability and Information Theory

06Method

How is one state's importance measured?

Causal tracing follows the causal-mediation idea of Vig and colleagues, a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior [7]. It uses three runs. In the clean run the prompt predicts the fact. In the corrupted run the subject is noised. In the corrupted-with-restoration run the network runs on the noisy embeddings except at one token $\overset{}{^}$ and layer $\hat{l}$ , where the clean state is written in; later computations run on whatever they receive. With $P$ for the probability of the true object's first token:

TE = P [o] - P_{*} [o], IE (\overset{}{^}, \hat{l}) = P_{*, clean h_{\overset{}{^}}^{(\hat{l})}} [o] - P_{*} [o]

The total effect TE is what the noise took away; the indirect effect IE of a state is what putting that one state back gives. The stage shows IE as a share of TE, from none to all, in the third map. Restoring the last token's state at the last layer returns everything by construction, and the recordings confirm it for every fact (test.mjs checks this). Averaging IE over facts gives the AIE the paper plots.

Here, for , restoring the last subject token's state () after block 8 returns a large share; the cells to its right and below show how that share changes with the layer and the token.

Go deeper: Activation Patching and Causal Tracing · DAGs, Confounders and Colliders

07Method

Why are MLP and attention restored in windows?

A single block's MLP write is small. The paper reports that restoring runs of ten values of the MLP's output (or attention's) over a window of layers gave results where single ones had "generally negligible effects, suggesting the decisive information is accumulated across layers". It plots each window at its centre layer. This piece restores a window of five writes (blocks $l - 2$ to $l + 2$ , clipped at the ends), which is the same fraction of depth as ten of 48; Hase and colleagues use a tracing window size of 5 by default on a 28-layer model. In an MLP window the clean write is put back at one token in each of the five blocks; attention and every other state are left to run noised.

One consequence is that a cell at layer 8 is not about layer 8: it is about blocks 6 to 10 together. A peak in the MLP trace therefore locates the writes to within a few layers at best, a point that matters when comparing with edits at a single layer.

Go deeper: Activation Patching and Causal Tracing

08The trace

Which single hidden states carry the answer?

On GPT-2 XL the paper found two regions: strongly causal states at the last subject token in the middle layers (the early site), and at the last token in the late layers, a late site whose presence the paper calls unsurprising. Its strongest single state was AIE 8.7% at layer 15 against an average total effect of 18.6% [1]. For on Qwen the highest state before the final token is , and the stage frames the map.

Read the map as a picture of sufficiency. In the average over the 72 facts, restoring the last subject token's state helps from the first layer, most in the first few layers, and by around layer 20 no longer helps at all. In the late layers it is the last token's states that restore the answer, which is where the answer is written. Tokens before the subject restore nothing, at any layer, because the noise never reached them. Whether that column at the subject's last token is as sharp for every fact is what the averages show.

Go deeper: The Residual Stream

09The trace

Do the MLPs or the attention heads carry it?

Restoring the MLP writes instead of the whole state asks whether the information at the subject was put there by an MLP. The paper's result for GPT-2 XL was that MLP contributions peak at AIE 6.6%, while attention at the last subject token is only 1.6% [1]. For the fact on the stage, ; .

Compare the MLP map with the hidden-state map: in the average the MLP windows matter at the subject's tokens in the first layers and again, more weakly, at the last token in the late layers, and almost nowhere else. That pattern is what the paper means by an early site and a late site. A single fact is noisy, and the next step compares it with the attention map for the same fact.

A share above 100% is possible: it means restoring that window gave the true object more probability than the clean run did, which the noise draws and the nonlinearity of the later layers allow.

Go deeper: Feed-forward and SwiGLU

10The trace

And attention?

For attention the paper found the opposite pattern: little at the subject, and more important at the last token of the prompt [1]. That is the late site: attention moves what the middle layers computed at the subject to the position that must write the answer. For the fact on the stage the strongest attention window is .

The two maps use the same colour scale, so their brightness compares directly. If the paper's picture holds in this model, the attention map should be dark at the subject and bright at the last token in the later layers, and the MLP map should be the other way round at the subject. The averages over all 72 facts test that, with a rule written down before they were computed.

Go deeper: Attention Mechanism · Multi-Query and Grouped-Query Attention

11The trace

What does the average look like?

The paper averaged over 1,000 prompts, 1,000 factual statements, and sorted positions into classes so that prompts of different lengths line up: the first subject token, the middle subject tokens, the last subject token, the first token after the subject, further tokens, and the final token. This piece adds a seventh, the tokens before the subject. A fact's class value is the mean over its tokens in the class, and the map is the mean over facts that have the class.

Read the figure by rows. The row for the last subject token is where the paper puts the early site; the row for the final token is its late site. Switching the restored module shows which of the two carries it. The figure can be restricted to set A, B or C, which is how the replication question is asked: if the three sets give three different pictures, one picture is not the answer.

Go deeper: Error Bars for Evaluations · Mechanistic Interpretability Primer

12The trace

Does the effect of a state depend on the MLPs after it?

A restored state might matter because later MLPs read it, or because later attention does. To separate the two the paper changes the graph: it collects each MLP's contribution in the corrupted run, then severs the MLP computations at token $i$ and freezes them in the baseline corrupted state while the clean state is put in, and compares. Its reading was that the lowest layers lose their causal effect without the activity of future MLP modules, while higher-layer states depend little on MLP activity, and that no such transition appears when attention is severed [1]. Note what kind of experiment this is: it is an intervention on the computation, not a reading of a map.

The stage draws three rows for the last subject token, averaged over the 72 facts: the restored hidden state as in the first trace, then the same with the MLPs of that token frozen at their noised values for every later block, then with its attention frozen. If the early states act through the MLPs after them, the middle row should be much lower than the top row at early layers.

Go deeper: Structural Causal Models and Counterfactuals

13The trace

What may be concluded from a map like this?

Three cautions come from the later literature. First, a restoration is sufficiency evidence only [8]. Second, the corruption matters: Zhang and Nanda found that on GPT-2 XL the clear peak around layer 16 under Gaussian noising is not salient at all under symmetric token replacement, with the noising peak 2 to 5 times higher [9], though they do not call the noising result illusory. Third, the paper's own conclusion from the maps is a statement about where information is carried in a forward pass, and Hase and colleagues argue that this is a different question from where to intervene to change what the model says (Causal Tracing answers a different question than model editing does) [2].

The authors of the first paper went further than the maps. They wrote that, because of earlier findings on exchanging layers, they conjecture that any fact could be equivalently stored in any one of the middle MLP layers. That sentence will matter in E2: it predicts that editing works about equally well anywhere in the middle, which is nearly what Hase and colleagues later reported, and it sits oddly beside the paper's own choice of one layer to edit.

Go deeper: Activation Patching and Causal Tracing · Circuit Tracing and Attribution Graphs

14Editing

If the facts are in the MLPs, what would storing one mean?

Geva and colleagues showed that feed-forward layers operate as key-value memories, where each key correlates with textual patterns and each value induces a distribution over the output vocabulary [10]. ROME takes a different cut of the same idea: the whole second matrix $W_{proj}$ is a linear associative memory. Any linear map $W$ can store pairs of keys $K = [k_{1} ∣ k_{2} ∣ \dots]$ and values $V = [v_{1} ∣ v_{2} ∣ \dots]$ by solving $W K \approx V$ , whose least-squares solution is $W = V K^{+}$ [1].

In Qwen the second matrix is the down-projection, 896 × 4,864, with no bias. Its input $k$ is the gated product (4,864 numbers) at a token; its output $Wk$ is the MLP's write (896 numbers) at that token. The stage draws the matrix for the layer you choose, mean-pooled 16 × 64 to 56 × 76 cells because 4.4 million cells will not fit on a table, with a pooled key above it and the pooled write $W k_{*}$ to its left.

Go deeper: Hopfield Networks and Attractor Memory · Dense Associative Memory · Linear Regression as Projection

15Editing

What is the key for "this subject"?

The key $k_{*}$ is the input of $W_{proj}$ at the last subject token, because the trace singled that position out. It depends on what precedes the subject, so it is averaged over contexts $x_{j}$ ending in the subject:

k_{*} = \frac{1}{N} j = 1 \sum N k (x_{j} + s), k (x) = σ (W_{fc} γ (a_{[x], i} + h_{[x], i}))

and the update uses the covariance of keys, $C = K K^{⊤}$ , estimated from text. Both choices are described twice, differently, in the paper [1]. The main text says it samples 50 random token sequences of length 2 to 10 from the model; the appendix says 20 texts, ten of length 5 and ten of length 10; the authors' released settings, two groups of ten prefixes, of lengths 5 and 10, match the appendix [11]. This piece uses three contexts: the bare prompt and two sampled prefixes (seed 4242). The covariance in the paper is 100,000 Wikipedia tokens; here it is 39,878 tokens of English prose from this repository's concept pages (code removed, the constant first position left out), a smaller and different corpus, and the matrix inverted with a ridge of 0.1% of its mean diagonal.

The stage shows the pooled key $k_{*}$ and below it $C^{- 1} k_{*}$ , the direction the update will write along. $C^{- 1}$ down-weights directions the model uses all the time, so that the edit does not disturb unrelated subjects.

Go deeper: Linear Regression as Projection

16Editing

What value should the memory return for the new fact?

The value $v_{*}$ is found by gradient descent on a vector $z$ added to the MLP's output at the last subject token:

L (z) = \frac{1}{N} j \sum - lo g P_{G (m_{i}^{(l^{*})} := z)} [o^{*} ∣ x_{j} + p] + D_{KL} (P_{G (m_{i^{'}}^{(l^{*})} := z)} [x ∣ p^{'}] P_{G} [x ∣ p^{'}])

The first term wants the new object; the second keeps the model's reading of "subject is a" (the prompt $p^{'}$ ) unchanged, to limit what the paper calls essence drift. The optimisation changes no weights; it finds a vector. The paper's appendix and the authors' released file disagree on the settings. The appendix: Adam with a learning rate of 0.5 and 1.5×10⁻³ weight decay, the KL factor 1×10², at most 20 steps. The released GPT-2 XL file: "v_weight_decay": 0.5 and "kl_factor": 0.0625 [11]. This piece follows the released file, with early stopping when the mean probability of the new object over the contexts reaches 0.95, and a cap of four times the norm of the original write on the size of $z$ 's change.

The stage's small chart is the search for the fact it was recorded on: the mean probability of the new object at each step.

Go deeper: Softmax and Cross-Entropy Gradient · Probability and Information Theory

17Editing

How does one matrix change by one rank?

Given the pair $(k_{*}, v_{*})$ , the matrix that stores it with the least disturbance to the other keys, in the least-squares sense with covariance $C$ , is a rank-one correction:

\hat{W} = W + Λ (C^{- 1} k_{*})^{⊤}, Λ = \frac{v _{*} - W k _{*}}{( C ^{- 1} k _{*} ) ^{⊤} k _{*}}

Read it in words. The correction is an outer product of a vector in the output space ( $Λ$ , 896 numbers, the part of $v_{*}$ the old matrix does not already produce) and a vector in the input space ( $C^{- 1} k_{*}$ , 4,864 numbers). Multiplying by $k_{*}$ gives $\hat{W} k_{*} = W k_{*} + Λ ((C^{- 1} k_{*})^{⊤} k_{*}) = v_{*}$ , so the new matrix returns exactly the new value for this key. Every other key is changed only through its overlap with $C^{- 1} k_{*}$ . The stage draws $W$ , the update and the sum, pooled, for the layer you choose below; the update has its own colour scale, because at the true scale it is a few per cent of $W$ and would not show.

The paper edits layer 18 of GPT-2 XL; the authors' released file for that model says layer 17, and for GPT-J says layer 5 where Hase and colleagues write that the default is layer 6 [2]. The released files number layers from 0, and the discrepancy is probably one of numbering, but neither document says so.

Go deeper: Locate-then-Edit Knowledge Editing · LoRA: Low-Rank Adaptation · Memory-based and Non-parametric Editing

18Editing

How do you know an edit worked?

An edit should change the answer for the prompt, for paraphrases of it, and for nothing else. Hase and colleagues use normalised versions of each metric that scale from 0 to 1 [2]. With $p$ and $p^{'}$ the probability of the new object's first token before and after the edit:

R = \frac{p ^{'} ( o ^{*} ∣ s , r ) - p ( o ^{*} ∣ s , r )}{1 - p ( o ^{*} ∣ s , r )}, P = the same on paraphrases, N = 1 - \frac{∣ p ^{'} ( o ^{*} ∣ s ^{*} , r ) - p ( o ^{*} ∣ s ^{*} , r ) ∣}{0.5 + ∣ p ( o ^{*} ∣ s ^{*} , r ) - 0.5∣}

$R$ (rewrite) asks whether the edited prompt now gives the new object; $P$ (paraphrase) whether two rephrasings of it do, with unrelated text in front; $N$ (neighbourhood) whether six prompts about similar subjects, whose answer should be unchanged, stay unchanged. Each is clipped to $[0, 1]$ . This piece adds one number, the overall score $O$ , the harmonic mean of the three, so that an edit that fails any one of them scores low. A harmonic mean is a choice made here, not by either paper.

Go deeper: Evaluating Knowledge Edits · Ripple Effects and Edit Portability

19Research desk

E1. Does the early site at the last subject token appear in a 24-block model?

Claim

Strongly causal states appear at an early site at the last token of the subject, carried by MLPs (AIE 6.6% against 1.6% for attention), with attention more important at the last token of the prompt [1]. Shown on GPT-2 XL, GPT-J, GPT-NeoX and smaller GPT-2 models; not on a model with Qwen2.5's gated MLP and grouped-query attention.

Hypotheses

H1a: in the MLP trace the highest cell short of the final token is the last subject token, at a middle layer (6 to 17). H1b: attention matters more at the final token than at the last subject token (ratio of maxima at least 2). H1c: at the last subject token MLPs carry more than attention (ratio at least 2). H1d (an intervention): freezing the token's MLPs removes the effect of restoring early states, and freezing attention does not.

Design

72 CounterFact facts in three disjoint sets of 24; hidden-state, MLP-window (5) and attention-window (5) traces at every token and layer; 2 noise draws each (seeds 7000 + 2 × case id + draw); noise 3σ; the severed traces at the last subject token. Rules fixed in research/README.md before the runs, with four amendments made before any result was opened.

Verdict rules

H1a: supported if in all three sets the highest cell outside the final token is the last subject token at layers 6–17; partly if the position holds in at least two sets. H1b, H1c: supported if the ratio is ≥ 2 in all three sets, partly if ≥ 1 in all. H1d: see the next step.

Reading it. The structure the paper describes is here: the MLP effect sits at the last subject token and not at the other positions, attention matters at the final token and almost not at the subject, and the pattern is the same in all three sets. What does not carry over is the depth. The paper's strongest single state was at layer 15 and its MLP trace peaked centred at the 17th layer of 48; here the peak is within the first few of 24 layers. The pre-registered band of 6 to 17 was a guess at "middle" and was missed for that reason, which is why H1a is only partly supported: the position prediction held in every set, the layer prediction did not. A window of five clipped at the start of the network means the peak cell covers blocks 0 to 4, so "layer 2" says the MLPs of the first few blocks, not one block.

Go deeper: Activation Patching and Causal Tracing · The Bootstrap and Resampling

20Research desk

E1 (continued). Does freezing the MLPs of the last subject token change what its states do?

Claim

The lowest layers lose their causal effect without the activity of future MLP modules; no such transition when attention is severed [1].

Hypothesis

H1d: freezing the MLPs of the last subject token (at their noised values, for every block after the restored one) removes most of the early layers' effect and little of the late layers'; freezing attention does not.

Verdict rule

With the pooled average effect summed over a band of layers: supported if MLP-frozen keeps at most 50% of the free effect over layers 0–7, attention-frozen keeps at least 75%, the MLP-frozen late-layer ratio is at least 75% (or the free late-layer sum is under 1%, which makes it untestable), and the MLP-frozen early ratio is below the attention-frozen one in every set. Partly if MLP-frozen keeps at most 75% and is at least 15 points below attention-frozen.

Reading it. This is the intervention behind E1's mechanism claim, and it agrees with the paper's: with the MLPs of the token frozen, most of what an early hidden state recovers disappears, and freezing attention changes almost nothing. In the late layers the state at the subject token matters little either way, so the paper's second condition could be tested only weakly; where it could, freezing the MLPs does not hurt. It shows that the early states act through the MLPs after them at that token. It does not show that the fact is stored there: a frozen module still leaves everything else free.

Go deeper: Structural Causal Models and Counterfactuals

21Research desk

E2. Is the layer where a fact traces the layer where editing it works best?

Claim

The layers at which edits generalize best correspond to the middle layers of the early site identified by Causal Tracing, with generalization peaking at the 18th layer [1]; against it, the assumption that one should change the knowledge in a model by editing the weights where it is stored turns out to be false [2]. GPT-2 XL against GPT-J; neither below 1.5B.

Hypotheses

H2a (Meng et al.): the edit layer with the highest mean overall score is within 3 layers of the layer where the MLP trace peaks at the last subject token. H2b (Hase et al.): once the layer is known, the tracing effect explains almost none of the variation in edit success. H2c: for a given fact, editing at the layer nearest to its own trace peak is no better than at the other layers.

Design

The first 12 facts of each set (36 facts) edited by ROME at eight layers, 1, 4, 7, 10, 13, 16, 19 and 22 (288 edits), scored by rewrite, paraphrase and neighbourhood and their harmonic mean. Three prompt sets are the replicates; the two prefix contexts were sampled once (seed 4242).

Verdict rules

H2a: supported if in each set the best edit layer and the traced peak layer are within 3; not supported if they differ by more than 3 in at least two sets. H2b: supported if R² of the layer alone is at least 0.30, the tracing effect adds less than 0.05, and the mean within-layer Spearman ρ is below 0.20 in magnitude; partly if two of the three. H2c: "no better" if the upper end of a 95% bootstrap interval over facts is below 0.05; "better" if the lower end is above 0.05.

Reading it. The edit works best in the first few blocks, where the trace peaks, and the two agree to within the three layers allowed, so H2a is supported. Two cautions come with that. The edit grid has a spacing of three layers, so "within 3" is a coarse test. And the edit score is not flat above the traced region: the rewrite score stays high until the last edit layer, as Hase and colleagues found for GPT-J, while paraphrase and neighbourhood scores are what move with the layer. An edit at the last layer rewrites the prompt without generalising to paraphrases, which is the regurgitation the paper itself warns about.

Go deeper: Locate-then-Edit Knowledge Editing · Evaluating Knowledge Edits

22Research desk

E2 (continued). Does a fact's own trace tell you where its edit will work?

Claim

The choice of layer explains almost all of the variance in rewrite score (94.7%) and the tracing effect is able to explain only 0.1% of the variance in edit success when accounting for the choice of edit layer [2], on GPT-J, n=652 facts it knows, where the correlation between tracing effect and ROME rewrite score at layer 6 was negative, not positive.

Hypotheses

H2b and H2c as above, on the 36 facts and 288 edits here.

Reading it. On Hase and colleagues' own outcome, the rewrite score, the result reproduces in kind: the layer explains most of the variance and the trace adds almost nothing. On the pre-registered outcome, the overall score, the layer explains much less, the trace still adds only a few per cent, and the within-layer correlation is negative: facts whose own trace is stronger at a layer were, if anything, edited slightly less well there. The paired comparison (H2c) came out positive as pre-registered, and the exploratory permutation says why: the traced layers are the layers where every fact edits best, and dealing them out at random gives nearly the same advantage. So at this scale the trace tells where editing works for the model as a whole, because the early blocks are where editing works, and tells nothing extra about an individual fact. That is consistent with the second paper's claim and does not overturn the first paper's average-level result.

Go deeper: Linear Regression as Projection · The Bootstrap and Resampling · Error Bars for Evaluations

23Research desk

What was found, in one table, and what is left open?

Lab note.

Open questions.

Does the picture survive a different corruption? Zhang and Nanda found the GPT-2 XL peak depends on it [9]; the same 72 facts could be traced with symmetric token replacement (swapping the subject for another subject) and the two compared at 0.5B.
Does it survive scale? The same code runs on Qwen2.5-1.5B (28 layers, width 1,536); a replication on the 1.5B model, which this run did not do, would show whether the early site moves with depth.
Is the early MLP site necessary as well as sufficient? Noising a state in the clean run (the other half of patching, in the vocabulary of Heimersheim and Nanda [8]) was not run.
Does editing at one layer "override" what is stored elsewhere, as Hase and colleagues suggest, and what does that cost? Editing several layers at once is what MEMIT does, scaling to thousands of associations [12]; the same scores could be computed for it.
Does the picture depend on the relation? 72 facts split across relations too thinly to say; a balanced set of a few relations would.

Go deeper: Sequential Editing and Model Collapse · Sparse Autoencoders for Interpretability

24Notes

What do the sources disagree about?

Go deeper: Ripple Effects and Edit Portability · Sequential Editing and Model Collapse

25Notes

What would be wrong to conclude from this page?

One model, 0.5B. Qwen2.5-0.5B-Instruct is an instruction-tuned model about a quarter the width of GPT-2 XL and half the depth. The research questions ask about 0.5B to 1.5B; only 0.5B was run.
72 facts, 36 edited. Sets of 24 and 12 are small; intervals are wide, and a verdict of "not supported" does not say the effect is absent.
Two noise draws, not ten, and one corruption method. The average traces are noisier than the paper's.
ROME is a probe, not a production editor. The paper says of it that it only edits a single fact at a time, and it is not intended as a practical method for large-scale model training; nothing here says how edits behave in sequence or at scale.
First-token probabilities. Multi-token answers are scored by their first token, and "known" means the model's top token matches it with probability at least 0.2.
Different covariance statistics. The edit's covariance comes from 39,878 tokens of this repository's prose, not 100,000 tokens of Wikipedia.
A prepended start token that the paper's models did not have.
Restoration is sufficiency evidence, and the page measures nothing about what happens when a state is damaged in the clean run.
The claim "the model stores the fact here" is not tested by anything on this page. What is tested is where restoring a state recovers an answer, and where editing a matrix changes it.

26Notes

How was this made, and how can it be checked?

The model was run in PyTorch 2.14 on a four-core CPU, float32, at the pinned revision, with a block-by-block forward pass written for this piece so that single states and writes could be replaced; A second runtime, Ollama (llama.cpp) serving the fp16 weights, agrees with PyTorch on the top token of all 72 prompts.

The research desk was pre-registered in research/README.md before the study runs, with the pipeline smoke test on a prompt outside CounterFact the only earlier run, and a lab note (research/LABNOTE.md) lists every deviation and amendment. research/summarise.mjs applies the verdict rules and writes the results prose from the numbers. The fact ledger's quotations come from the arXiv LaTeX sources, the authors' released hyper-parameter files and the model card, and were grep-checked.

Cite as: this page, Locating a fact (sigmoid, 2026); the anchor papers are [1] and [2].

References

Meng, K., Bau, D., Andonian, A. and Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022 (arXiv:2202.05262) Anchor paper: causal tracing and ROME. Quotes are from the arXiv LaTeX source fetched 2026-10-21 (the current arXiv version).
Hase, P., Bansal, M., Kim, B. and Ghandeharioun, A. (2023). Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. NeurIPS 2023 (arXiv:2301.04213) Anchor paper: the dispute. Quotes are from the arXiv LaTeX source fetched 2026-10-21.
Qwen Team (2024). Model card (README.md) of Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576. Hugging Face model repository Layer count, heads, parameter count of the model on this page.
Qwen Team (2024). config.json of Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576. Hugging Face model repository Widths, vocabulary and tying of the model on this page.
Yang, A. et al. (Qwen Team) (2024). Qwen2.5 Technical Report. arXiv:2412.15115 The model on this page. Quotes from the arXiv LaTeX source.
OpenAI (Radford, A. et al.) (2019). config.json of openai-community/gpt2-xl at revision 15ea56d. Hugging Face model repository Widths of the model the ROME paper studied most; the paper itself gives only the layer count and 1.5B.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y. and Shieber, S. (2020). Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS 2020 (arXiv:2004.12265) The causal-mediation framing that causal tracing takes from. Quote from the arXiv LaTeX source.
Heimersheim, S. and Nanda, N. (2024). How to use and interpret activation patching. arXiv:2404.15255 Denoising versus noising, sufficiency versus necessity. Quotes from the arXiv LaTeX source.
Zhang, F. and Nanda, N. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024 (arXiv:2309.16042) Shows that the choice of corruption method (Gaussian noising or symmetric token replacement) changes the traced picture. Quotes from the arXiv LaTeX source.
Geva, M., Schuster, R., Berant, J. and Levy, O. (2021). Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP 2021 (arXiv:2012.14913) The key-value view of the MLP that ROME builds on. Quote from the arXiv LaTeX source.
Meng, K. et al. (2022). hparams/ROME/gpt2-xl.json and EleutherAI_gpt-j-6B.json in github.com/kmeng01/rome (main, fetched 2026-10-21). GitHub The authors' released hyper-parameters, which differ from the numbers in the paper's appendix.
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y. and Bau, D. (2023). Mass-Editing Memory in a Transformer. ICLR 2023 (arXiv:2210.07229) MEMIT, the multi-layer, many-fact successor of ROME. Quote from the arXiv LaTeX source.
Niu, J., Liu, A., Zhu, Z. and Penn, G. (2024). What does the Knowledge Neuron Thesis Have to do with Knowledge?. ICLR 2024 (arXiv:2405.02421) A later reassessment of the claim that MLPs store knowledge. Quote from the arXiv abstract in the LaTeX source.

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
last subject token	the early site	[1] §2.2 but their emergence at an early site at the last token of the subject is a new discovery
MLP, middle layers, last subject token	the localized factual association hypothesis	[1] §2.3 This hypothesis localizes factual association along three dimensions, placing it (i) in the MLP modules (ii) at specific middle layers (iii) and specifically at the processing of the subject's last token.
24	blocks	[3] Model card Number of Layers: 24
14 query, 2 key-value	attention heads	[3] Model card Number of Attention Heads (GQA): 14 for Q and 2 for KV
RoPE, SwiGLU, RMSNorm, QKV bias, tied embeddings	architecture	[3] Model card Architecture: transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
0.49B (0.36B without embeddings)	parameters	[3] Model card Number of Parameters: 0.49B
0.36B	non-embedding parameters	[3] Model card Number of Paramaters (Non-Embedding): 0.36B
896	residual width	[4] config.json "hidden_size": 896
4,864	MLP width	[4] config.json "intermediate_size": 4864
151,936	vocabulary rows	[4] config.json "vocab_size": 151936
true	tied embeddings	[4] config.json "tie_word_embeddings": true
18 trillion tokens	pre-training data	[5] Abstract we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens
1.5B parameters	GPT-2 XL	[1] §2.2 plots the AIE of the internal components of GPT-2 XL (1.5B parameters)
48	layers of GPT-2 XL	[1] Appendix A.3 fewer layers (44 and 28 layers instead of 48)
1,600	residual width of GPT-2 XL	[6] config.json "n_embd": 1600
25	attention heads of GPT-2 XL	[6] config.json "n_head": 25
21,919 records	size of CounterFact	[1] §4.2 Containing 21,919 records with a diverse set of subjects, relations, and linguistic variations
32.6%	GPT-J completion accuracy on CounterFact	[2] Appendix A GPT-J achieves a completion accuracy of 32.6% under this scheme
151643	beginning-of-sequence id	[4] config.json "bos_token_id": 151643
3 times the embedding standard deviation	corruption noise	[1] §2.1 to be 3 times larger than the empirical standard deviation of embeddings
sigma = 0.094	noise in Hase et al.	[2] §3.2 we add Gaussian noise with \sigma=0.094 to the token embeddings of s
ten	noise draws per prompt	[1] Appendix A.1 the process is repeated ten times with different samples of corruption noise
27.0%	probability of the true object on the 1,000 known facts	[1] Appendix A.1 the predicted probability of the correct object token calculated by GPT-2 XL averages 27.0%
8.47%	after corruption	[1] Appendix A.1 On average, this reduces the correct object token score to 8.47%
the framework	causal mediation analysis	[7] Abstract We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior.
sufficiency	what restoration tests	[8] §2 Denoising (a clean → corrupt patch) shows whether the patched activations were sufficient to restore the model behaviour.
not complements	denoising and noising	[8] §4 These two are usually not complements of each other, nor does one imply the other.
ten layers	window for MLP and attention traces	[1] Appendix A.2 we restored runs of ten values of
5	tracing window in Hase et al.	[2] §4.1 we use a tracing window size of 5 by default
unsurprising	the late site	[1] §2.2 The presence of strong causal states at a late site immediately before the prediction is unsurprising
AIE 8.7% at layer 15	strongest single state	[1] §2.2 AIE=8.7% at layer 15
18.6%	average total effect, GPT-2 XL	[1] §2.2 The ATE of this experiment is 18.6%
6.6% against 1.6%	MLP against attention at the last subject token	[1] §2.2 MLP contributions peak at AIE 6.6%, while attention at the last subject token is only AIE 1.6%
more important there	attention at the last token	[1] §2.2 attention is more important at the last token of the prompt
1,000	facts in the average trace	[1] §2.2 We compute the average indirect effect (AIE) over 1000 factual statements
lowest layers lose their effect	severing the MLPs	[1] §2.2 the lowest layers lose their causal effect without the activity of future MLP modules
GN against STR	a different corruption moves the peak	[9] §4 Notice that the clear peak around layer 16 under GN is not salient at all under STR.
2 to 5 times higher	size of the Gaussian-noising peak	[9] §4 we find the peak value of GN to be 2×–5× higher than STR
a different question	their conclusion	[2] §7 Causal Tracing answers a different question than model editing does
any middle MLP layer	the conjecture about layers	[1] §2.3 We conjecture that any fact could be equivalently stored in any one of the middle MLP layers.
noisy and uninformative	the direct effect	[1] §2.1, footnote we found this effect to be noisy and uninformative
the view ROME builds on	MLPs as key-value memories	[10] Abstract feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary
50 sequences of length 2 to 10	random prefixes for k*, in the main text	[1] §3.1 we sample x_j by generating 50 random token sequences of length 2 to 10 using G
20 texts: ten of length 5, ten of length 10	random prefixes for k*, in the appendix	[1] Appendix C.2 We sample 20 texts to compute the prefix
[[5, 10], [10, 10]]	prefix lengths in the released code	[11] hparams/ROME/gpt2-xl.json "context_template_length_params": [[5, 10], [10, 10]]
100,000 Wikipedia tokens	second-moment statistics	[1] Appendix C.2 computed using 100,000 samples of hidden states
1.5e-3	weight decay when solving for v*, per the paper	[1] Appendix C.2 solved for using Adam with a learning rate of 0.5 and 1.5×10⁻³ weight decay
1e2	KL scaling factor, per the paper	[1] Appendix C.2 The KL divergence scaling factor … is set to 1×10².
at most 20	steps to find v*	[1] Appendix C.2 The minimization loop is run for a maximum of 20 steps
0.5	weight decay in the released code	[11] hparams/ROME/gpt2-xl.json "v_weight_decay": 0.5
0.0625	KL factor in the released code	[11] hparams/ROME/gpt2-xl.json "kl_factor": 0.0625
layer 18	the layer ROME edits	[1] Appendix C.2 We perform the intervention at layer 18.
17	edit layer in the released code (GPT-2 XL)	[11] hparams/ROME/gpt2-xl.json "layers": [ 17 ]
5	edit layer in the released code (GPT-J)	[11] hparams/ROME/EleutherAI_gpt-j-6B.json "layers": [ 5 ]
normalised to 0 to 1	the scores Hase et al. use	[2] §3.4 we use normalized versions of each metric that we design to scale from 0 to 1
centred at the 17th layer	where the paper's MLP trace peaks	[1] Appendix A.2 finds the maximum effect at an interval at the last entity token, centered at the the 17th layer
generalisation peaks at layer 18	edits against the trace	[1] §4.3 the layers at which edits generalize best correspond to the middle layers of the early site identified by Causal Tracing
layer 18	where ROME edits generalise best (GPT-2 XL)	[1] §4.3 with generalization peaking at the 18th layer
editing where it is stored is not better	the headline	[2] §1 the assumption that one should change the knowledge in a model by editing the weights where it is stored turns out to be false
94.7%	variance in rewrite score explained by the edit layer	[2] §4.3 the choice of layer explains almost all of the variance in rewrite score (94.7%)
0.1%	what the tracing effect adds	[2] §4.3 the tracing effect is able to explain only 0.1% of the variance in edit success when accounting for the choice of edit layer
652	facts GPT-J knows, in Hase et al.	[2] §4.2 Our final sample size is n=652.
-0.13	correlation between tracing effect and ROME rewrite score at layer 6	[2] §4.3 there is not a positive relationship but a negative relationship between the rewrite score and the tracing effect
thousands of associations	MEMIT	[12] Abstract demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B)
many layers could store a fact	their reading	[2] §7 Many layers could store a fact, and it happens that some do.
3% of the variance	the one case where tracing helps	[2] §5 tracing effects explain an additional 3% of the variance in edit success
at best an oversimplification	the knowledge neuron thesis	[13] Abstract We find that this thesis is, at best, an oversimplification.
models larger than GPT-J	a limit Hase et al. state	[2] Limitations the conclusions from our analysis may not generalize to models larger than GPT-J (6B parameters)
one fact at a time	what ROME is for	[1] §4.6 it only edits a single fact at a time, and it is not intended as a practical method for large-scale model training

How this was made

Recorded from the real model (Qwen/Qwen2.5-0.5B-Instruct at revision 7ae5576, float32 PyTorch on CPU): for 72 facts the probability of the true object under the clean run, the noised run (two draws) and every restoration, plus state norms, and for 288 edits the scores. Computed in the page: every indirect effect and share, the token-class averages, the E1 ratios and verdicts, regressions, Spearman correlations, the bootstrap and permutation tests, and the rank-one algebra on the shipped vectors. test.mjs checks the engine against numpy and SciPy from the raw results, the recordings against a second route (library forward with hooks, 54 cells) and a second runtime (Ollama, all 72 prompts, same top-1 token), the method's invariants, the edit scores recomputed from stored probabilities, the parameter count, and every static number in the story. Quotations come from the arXiv LaTeX sources, the released hyper-parameter files and the model card, and were grep-checked. The research desk is pre-registered in research/README.md with a lab note.

Built with the dissection skill. The model was run on a CPU; every recording and test is in probe/ and research/.

Cite this page

@misc{sigmoid-locating-a-fact,
  title  = {Locating a fact},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/locating-a-fact}
}