Editing Facts Into Weights: Why Knowledge Editing Keeps Losing to Retrieval

In 2022 a paper showed you could make GPT-2 XL claim the Eiffel Tower stands in Rome by changing one matrix, with a closed-form update computed in about a second, no gradient descent over the network, no retraining run. Ask the model where the Eiffel Tower is and it says Rome. Ask it what you can see from the top and it describes the view over Rome.

That demonstration launched a research programme with an obvious industrial motive. Models go stale the moment pretraining ends. Companies are renamed, laws change, someone's job title changes, a model memorised a wrong fact from a bad web page. Retraining to fix one sentence is absurd; fine-tuning on a single fact is unreliable and expensive. A one-second surgical edit is exactly the operation a deployment team wants.

Four years of benchmarks later, the verdict is unusually clear for this field. Edits take, and they do not travel. They degrade the model when you apply thousands of them in sequence. They do not update the consequences of the fact they insert. And on the largest realistic benchmark yet built, an ordinary retrieval-augmented baseline beats every editing method on nearly every axis that matters (Thede et al., 2025, WikiBigEdit, ICML 2025, arXiv:2503.05683).

Why this matters: Every team running an LLM in production eventually asks how to correct a fact without retraining. Weight editing is the answer that looks cheapest and is almost never right. Knowing precisely why it fails tells you what to build instead, and tells you the narrow conditions under which editing is still the better tool.

TL;DR

• ROME changes a fact with a rank-one update to one MLP matrix, computed in closed form from a constrained least-squares problem. MEMIT spreads the same update across several layers and scales to 10,000 edits in a single pass, reporting an editing score of 85.8 on CounterFact where ROME gets 50.3 and MEND 23.1.
• The localisation argument that motivated the method does not hold up: causal-tracing signal at a layer is statistically uncorrelated with whether an edit at that layer succeeds (Hase et al., 2023).
• Applied sequentially, edits produce gradual forgetting followed by abrupt collapse, with the catastrophic phase for MEMIT reported around 1,400 edits (Gupta et al., 2024).
• Edits do not propagate to entailed facts. On the RippleEdits benchmark, a plain in-context baseline scores best, beating every parametric method (Cohen et al., 2024, TACL).
• Efficacy is the metric everyone reports and the one that means least. Measured on real hallucinations across five axes, performance beyond efficacy is "generally unsatisfactory" for all methods (Huang et al., 2025, ICLR).
• At 500,000 real Wikidata updates, retrieval augmentation leads on every axis except multi-hop reasoning, where composing several retrieved edits is still weak. That exception is the whole remaining argument for parametric methods.
• Part of the collapse literature was an implementation artefact: r-ROME produces parameter changes orders of magnitude smaller and does not collapse (Gupta, Baskaran & Anumanchipalli, 2024). The scaling limit survived the fix; the cliff moved.

At a Glance

flowchart LR
    F["New fact arrives"] --> D{"Where does it live?"}
    D -->|"weights"| E["Rank-one edit<br/>ROME, MEMIT"]
    D -->|"side memory"| M["Edit store plus scope check<br/>SERAC, GRACE, WISE"]
    D -->|"context"| R["Retrieve into the prompt"]
    D -->|"training set"| T["Continual finetune or retrain"]
    E --> EO["Free at inference<br/>degrades with edit count"]
    M --> MO["Reversible<br/>scope classifier is the risk"]
    R --> RO["Propagates through reasoning<br/>costs tokens per request"]
    T --> TO["Durable<br/>slow and expensive"]
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    class F blue
    class D slate
    class E,M purple
    class R,T teal
    class EO amber
    class MO,RO,TO teal

The four lanes differ in one respect that predicts almost everything else: whether the new fact ends up somewhere every circuit in the forward pass can read, or somewhere only one matched key can reach.

Before Model Editing: Fine-Tuning and Its Failures

The pre-history is short and the problem is old. A deployed model holds a wrong fact. Fine-tuning on the corrected statement works badly for reasons that are easy to state and hard to fix: a single example gives a tiny, noisy gradient signal, so the learning rate has to be raised, and raising it drags unrelated behaviour along. Train harder and the model overfits the exact phrasing. Train softer and the correction does not stick.

The first generation of editors attacked this by learning how to edit. KnowledgeEditor and MEND train auxiliary networks that transform the fine-tuning gradient into a targeted weight change; MEND does it with a low-rank decomposition of the gradient, which makes the transformation tractable even for models above ten billion parameters and trainable on one GPU in under a day (Mitchell et al., 2022, Fast Model Editing at Scale, ICLR 2022, arXiv:2110.11309).

The second generation asked a different question, borrowed from interpretability. If feed-forward layers act as key-value memories, with each key matching an input pattern and each value inducing a distribution over output tokens (Geva et al., 2021, EMNLP), then a fact has an address. Find it, overwrite it, done. No learned editor, no gradient steps, just linear algebra.

timeline
    title From key-value memories to lifelong editing
    2021 : Geva et al. show feed-forward layers act as key-value memories
         : Mitchell et al. train MEND to transform fine-tuning gradients into edits
    2022 : ROME edits a fact with one rank-one update guided by causal tracing
         : SERAC keeps edits in an explicit memory and leaves weights frozen
    2023 : MEMIT spreads updates over critical layers and reaches 10,000 edits
         : Hase et al. show causal tracing does not predict where editing works
         : MQuAKE shows edited models fail multi-hop questions entailed by the edit
    2024 : RippleEdits finds in-context editing beats parametric methods
         : Gupta et al. document gradual then catastrophic forgetting under sequential edits
         : r-ROME shows some model collapse was an implementation artefact
    2025 : AlphaEdit projects updates onto the null space of preserved knowledge
         : HalluEditBench evaluates edits on real hallucinations across five axes
         : WikiBigEdit finds retrieval beats every editor over 500,000 real updates
    2026 : Continuous-drift benchmarks push both editing and vanilla RAG past their limits

[IMAGE: Two-panel schematic of an MLP block. Left panel: the up-projection acting as a bank of pattern detectors, with one key highlighted for the input "The Eiffel Tower is located in". Right panel: the down-projection as a bank of value vectors, with the matched value pushing probability toward "Paris". Caption: "The key-value reading of feed-forward layers is the premise the whole editing literature is built on."]

How Locate-Then-Edit Actually Works

Three steps: find the layer, decide what the new output vector should be, and solve for the smallest weight change that produces it.

Finding the layer with causal tracing

Run the prompt normally and record the hidden states. Run it again with the subject tokens corrupted by noise, so the model can no longer identify what it is being asked about, and the correct answer's probability collapses. Now restore individual hidden states from the clean run into the corrupted run, one at a time, and measure how much of the correct prediction comes back. States whose restoration recovers the answer are causally decisive for it.

In GPT-style models this procedure produces a consistent picture: a strong effect at mid-layer MLP modules at the last subject token (Meng et al., 2022, Locating and Editing Factual Associations in GPT, NeurIPS 2022, arXiv:2202.05262). That site became the default edit target.

It should not have. Hase et al. ran the obvious control, editing at every layer and comparing edit success against the tracing signal, and found no correlation. Facts can be changed by editing weights in a different location from where tracing points, and tracing "does not provide any insight into which model MLP layer would be best to edit" (Hase et al., 2023, Does Localization Inform Editing?, arXiv:2301.04213). The method works; the story about why it works does not. That gap is worth holding onto, because a technique whose mechanism is not understood tends to fail in ways its designers did not predict, which is what happened next.

[IMAGE: Causal tracing heatmap, layers on the y-axis and token positions on the x-axis, with a bright band at mid-layer MLP modules over the last subject token and a second weaker band at late attention layers over the final token. Caption: "The picture that launched locate-then-edit, and that later turned out not to predict where editing works."]

Choosing the new value vector

The key \(k_*\) is the MLP's internal activation for the subject at the chosen layer, averaged over a few prompt templates so it is not tied to one phrasing. The value \(v_*\) is found by optimisation, not by hand: run a short gradient search over the vector that, when substituted at that position, maximises the probability of the target token, plus a KL term that keeps the subject's other associations from drifting. A few dozen steps on one vector, not on the model.

Two consequences follow immediately, and both show up later as failure modes. The objective is defined over a handful of prompt templates, so nothing forces the edit to fire on phrasings outside them. And the KL term constrains the subject's own distribution, not the distribution of any other subject that happens to sit nearby in key space.

Solving for the weight change

Treat the down-projection \(W\) as an associative memory mapping a matrix of keys to a matrix of values, \(WK \approx V\). Insert the new pair while disturbing the rest as little as possible:

The Lagrangian has a closed-form solution, and it is rank one:

Here \(C = \mathbb{E}[kk^{\top}]\) is the uncentered second moment of MLP keys, precomputed once from a sample of ordinary text. The numerator carries the error the edit must correct, \(v_* - Wk_*\), and the denominator normalises so the constraint is met exactly. Write it as \(\Delta = u\tilde{k}^{\top}\) with \(u = v_* - Wk_*\) and \(\tilde{k} = C^{-1}k_* / ((C^{-1}k_*)^{\top}k_*)\), and the entire behaviour of the method is visible in one line: for any key \(k\), the output changes by \(u\,(\tilde{k}^{\top}k)\).

Orthogonal keys are untouched. Overlapping keys get a proportional dose of the same shift, in the same direction, whether or not that direction is right for them. There is no term in the objective that knows the difference between a neighbour that should change and a neighbour that should not.

MEMIT: the same solve, spread out

MEMIT keeps the objective and changes the distribution of the update. Rather than forcing the whole residual into one matrix, it computes the required change at the final critical layer and spreads it across a range of earlier critical layers, each absorbing a fraction. Because no single matrix takes the full hit, the method survives batch sizes that break ROME: thousands of associations at once for GPT-J (6B) and GPT-NeoX (20B), reported as exceeding prior work by orders of magnitude (Meng et al., 2023, Mass-Editing Memory in a Transformer, ICLR 2023, arXiv:2210.07229).

[IMAGE: Diagram of the MEMIT update distribution. A required residual vector at the final critical layer, split into fractional contributions written back into layers L-4 through L, with arrow widths proportional to the share each layer absorbs. Caption: "MEMIT's scaling advantage is a load-balancing trick, not a different objective."]

Seeing It in Motion

The edit pipeline, end to end:

flowchart TB
    P["Prompt: subject plus relation"] --> CT["Causal trace<br/>corrupt and restore"]
    CT --> L["Pick critical layer"]
    P --> K["Compute key k star<br/>average over templates"]
    K --> V["Optimise value v star<br/>target token plus KL term"]
    L --> S["Closed-form solve"]
    K --> S
    V --> S
    S --> W["Write rank-one delta<br/>into MLP matrix"]
    W --> A["Accumulated drift<br/>grows with edit count"]
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class P blue
    class CT,K,V,L purple
    class S,W teal
    class A amber

At inference the two designs diverge completely. The edited model answers from parameters and pays nothing extra; the retrieval design answers from context and pays on every request:

sequenceDiagram
    participant U as User
    participant G as Gateway
    participant S as Edit store
    participant M as Model
    U->>G: Who owns Company X now?
    alt Weight-edited model
        G->>M: prompt only
        M-->>U: answer from edited parameters
        Note over M: zero added latency<br/>fires only on matched keys
    else Retrieval augmented
        G->>S: search for facts about Company X
        S-->>G: 2 updated facts, 180 tokens
        G->>M: prompt plus retrieved facts
        M-->>U: answer grounded in context
        Note over G,M: added retrieval hop<br/>plus prefill on every request
    end

And the lifecycle of an edited checkpoint, which is the part teams discover late:

stateDiagram-v2
    [*] --> Base
    Base --> Edited: apply edit n
    Edited --> Edited: apply edit n plus 1
    Edited --> Drifting: previously edited facts start to fade
    Drifting --> Collapsed: downstream ability falls away
    Drifting --> Base: restore checkpoint, replay log
    Collapsed --> Base: restore checkpoint, replay log
    Collapsed --> [*]: discard

There is no edge from Collapsed back to Edited. Weight edits compose destructively, so the only recovery is the base checkpoint plus an ordered log you can replay without the offending edit. Teams that did not keep the log discover that "roll back the third edit" is not an operation they have.

Watch It Run

By the Numbers

Sources: all figures are from the linked papers. The 1,400-edit onset is a measured value for one model and dataset combination, not a universal constant; the same paper reports ROME entering its catastrophic phase considerably earlier. Percentages from WikiBigEdit are single-configuration results and should be read as the shape of the gap, not as a specification.

A Concrete Example

Take the machinery down to two dimensions so the arithmetic is checkable by hand. Assume whitened keys, so \(C = I\) and \(\tilde{k} = k_*\).

Step 1. The current state. The key for (Eiffel Tower, is located in) is \(k_* = (0.6,\ 0.8)\), unit length. The model's current output vector at that layer is \(Wk_* = (1.0,\ 0.0)\), which decodes to "Paris".

Step 2. The target. We want \(v_* = (0.2,\ 0.9)\), which decodes to "Rome". The error the edit must carry is

Step 3. The update. With \(C = I\) the denominator is \(k_*^{\top}k_* = 1\), so

Step 4. Check the constraint. \(\Delta k_* = u\,(k_*^{\top}k_*) = u\), so the new output is \(Wk_* + u = v_*\) exactly. Efficacy: 100%. This is the number papers headline.

Step 5. An unrelated fact. Take a key with no overlap, \(k_2 = (0.8,\ -0.6)\). Then \(k_*^{\top}k_2 = 0.48 - 0.48 = 0\), so the output shift is zero. Locality on random facts: perfect. This is why locality measured on randomly sampled prompts is nearly free and tells you almost nothing.

Step 6. A near neighbour. Now take (Louvre, is located in), whose key points in almost the same direction because the relation and the subject type are shared: \(k_1 = (0.5,\ 0.866)\). The overlap is \(0.6(0.5) + 0.8(0.866) = 0.993\), so the Louvre's output moves by \(0.993\,u \approx (-0.79,\ 0.89)\), landing at approximately \((0.21,\ 0.89)\). The Louvre is now in Rome too. Nothing in the objective objected, because the covariance term controls the average key, and this key is not average, it is adjacent.

Step 7. An entailed fact, updated wrongly. Take (Eiffel Tower, is in the country), with key \(k_3 = (0.1,\ 0.995)\). Overlap is \(0.06 + 0.796 = 0.856\), so this output also moves by \(0.856\,u\), in the direction of the token "Rome". The correct entailed answer is "Italy". The edit does not leave the entailed fact alone and it does not update it correctly either: it pushes it toward the wrong token with 86% of the force of the original edit. Ripple failure is not silence, it is noise.

[IMAGE: Two-dimensional vector plot showing the unit circle, the edit direction k star at (0.6, 0.8), the neighbour key k1 at (0.5, 0.866) almost on top of it, the entailment key k3 at (0.1, 0.995), and the orthogonal key k2 at (0.8, -0.6); arrows of length proportional to overlap show the shift each one receives. Caption: "Locality is free for the orthogonal key and impossible for the adjacent one."]

Step 8. Accumulation. Each edit contributes a matrix of Frobenius norm \(\|u\|\cdot\|k_*\| = 1.20\). Apply \(n\) edits whose directions are mutually uncorrelated and the accumulated perturbation grows roughly as \(1.20\sqrt{n}\): about 13 at 100 edits, about 45 at 1,400. Real edit directions are correlated, which pushes growth toward linear. Meanwhile every individual edit computed its preservation term against the original key covariance, as though it were the only edit ever applied. That mismatch, compounding, is the mechanism behind the measured collapse.

[IMAGE: Line chart with edit count on a log x-axis from 1 to 10,000 and two y-series: accumulated perturbation norm (rising as sqrt of n, then faster) and downstream task accuracy (flat, then a slow decline, then a cliff). A vertical marker at 1,400. Caption: "Gradual forgetting is the region where the two curves diverge quietly; the cliff is where they meet."]

Where It Breaks

Efficacy is measured where the edit was made

Every editing paper reports how often the edited prompt now yields the new answer, and the numbers are near-perfect because that is what the optimisation targeted. HalluEditBench evaluated methods across efficacy, generalisation, portability, locality and robustness on more than 6,000 hallucinations that the target models genuinely produce, and found performance beyond efficacy "generally unsatisfactory" across methods (Huang et al., 2025, ICLR 2025, arXiv:2410.16251).

The same paper exposes a quieter problem with the standard datasets: they do not verify that the model was wrong before the edit. If the model already knew the answer, a successful edit measures nothing at all.

Consequences do not propagate

Inject "Jack Depp's father is Johnny Depp" and you have implied a sibling relation, a grandparent relation and a child relation. RippleEdits tests exactly this, across 5,000 edits and six criteria including logical generalisation, compositionality, subject aliasing and relation specificity, and finds that editing methods fail to make consistent changes. The in-context baseline wins (Cohen et al., 2024, TACL 12:283-298).

MQuAKE reaches the same conclusion from the reasoning side: edited models recall the edited fact and then fail multi-hop questions whose answers follow from it, with accuracy degrading as hops go from two to three to four, and degrading further when several edits interact (Zhong et al., 2023, EMNLP 2023, arXiv:2305.14795). Its proposed fix stores edited facts externally and prompts iteratively, which is a retrieval architecture wearing an editing paper's title.

Sequential application degrades the model

Batch editing and sequential editing are different experiments, and the literature's headline numbers come from the first while deployments live in the second. Applied one after another, edits produce gradual forgetting of earlier edits and of downstream ability, then an abrupt catastrophic phase, reported at roughly 1,400 edits for MEMIT (Gupta et al., 2024, ACL Findings, arXiv:2401.07453).

The honest complication: some of this was an artefact. Certain "disabling edits" caused instant collapse because of irregularities in the original ROME implementation, appearing with CounterFact but not zsRE, and the corrected r-ROME produces parameter changes orders of magnitude smaller that grow smoothly and do not collapse under large-scale sequential editing (Gupta, Baskaran & Anumanchipalli, 2024, arXiv:2403.07175). The cliff moved. It did not disappear: on real-world updates at scale, r-ROME degrades along with the rest.

Geometry buys headroom, not immunity

AlphaEdit is the strongest mitigation to date. It projects each perturbation onto the null space of the preserved knowledge's key matrix, so the update is orthogonal to the directions in which retained facts are represented, lifting locate-then-edit performance by about 36% on average across GPT2-XL, GPT-J and LLaMA3 (Fang et al., 2025, ICLR 2025 outstanding paper, arXiv:2410.02355). The technique is elegant and the limit is arithmetic: the null space has finite dimension, and every edit spends some of it.

The ranking inverts at scale

WikiBigEdit built 500,000 question-answer pairs from real Wikidata changes rather than synthetic counterfactuals, and compared editing against retrieval augmentation and continual fine-tuning over that stream. Locate-then-edit methods collapse within the first 10,000 updates. Methods designed for high edit volumes converge back toward pre-edit accuracy. An efficiently implemented retrieval-augmented baseline outperforms every editing method on all axes except multi-hop reasoning, holding around 78.6% on rephrasings and 66.2% under persona conditioning, with no locality penalty at all because no weights change (Thede et al., 2025, ICML 2025, arXiv:2503.05683).

A method comparison run at 100 edits will recommend the wrong architecture for a system that will accumulate 100,000.

[IMAGE: Grouped bar chart of five evaluation axes (update accuracy, rephrase, locality, personas, multi-hop) comparing a locate-then-edit method, a high-volume editing method and a retrieval baseline at 10k+ updates, with the retrieval bar highest everywhere except multi-hop. Caption: "The one bar where editing-adjacent approaches still compete is composition over several updates."]

Alternative Designs

[IMAGE: Positioning chart with expected lifetime edit volume on the x-axis (log scale, 10 to 1,000,000) and added cost per request on the y-axis. Locate-then-edit sits bottom-left, memory-based editing mid-left, in-context editing top-left, retrieval augmentation mid-right, continual fine-tuning and retraining bottom-right. A shaded band marks where sequential collapse has been measured. Caption: "Pick the lane by expected edit volume first; everything else is a second-order consideration."]

The comparison that decides most real cases is the second-to-last row against the fifth. Retrieval and fine-tuning are complements on different clocks: retrieval keeps you current between checkpoints, and consolidation folds the accumulated corrections into weights when the next training run happens anyway. Editing sits awkwardly between them, cheaper than either and durable in neither sense.

How It Is Used in Practice

Almost nobody ships sequential weight editing as the maintenance path for a production model, and the reasons are operational before they are scientific. An edited checkpoint is a fork: it is not the vendor's model, it does not match the evaluation suite that was run on the base, and its provenance is an ordered list of mutations rather than a version. Serving stacks assume weights are immutable artefacts identified by a hash. Editing breaks that assumption for a benefit that a retrieval index provides reversibly.

Where editing does earn its place:

Pre-release correction of a small, fixed set. A model about to ship holds a handful of known-wrong associations. Twenty edits, applied once, evaluated as a batch against a locality set and a general benchmark, then frozen as a new artefact with its own hash. This is the regime the batch numbers describe, and it works.

Safety and unlearning-adjacent surgery. When the requirement is that a capability or association should not be reachable through the weights, retrieval does not help, because the parametric fact is exactly the thing at issue.

Research on mechanism. Editing remains one of the sharpest causal probes available for asking where a behaviour lives, even after the localisation result showed the probe and the intervention answer different questions.

Everywhere else, the architecture that has won is a retrieval layer over a curated corpus of corrections, with the model instructed to prefer retrieved facts over memorised ones, and a consolidation pass that folds the corpus into the next training run. The 2026 literature is already pushing on the remaining weakness of that design, temporal consistency: vanilla RAG and learning-based adaptation both struggle when knowledge drifts continuously rather than in discrete corrections, because a retrieved current fact and a memorised past fact can both be right at different times and the model has no principled way to order them (RAG or Learning?, ACL Findings 2026, arXiv:2604.05096).

[IMAGE: Operational diagram of the pattern that actually ships. A corrections corpus with an approval workflow feeds a retrieval index; the index serves a frozen model behind a gateway; a scheduled consolidation job folds the corpus into the next fine-tune, after which entries can be retired from the index. Caption: "Corrections live in a corpus with a lifecycle, not in a matrix."]

Insights Worth Remembering

1. A rank-one edit is a projection, not a belief update. The change applies to every key in proportion to its overlap with one direction. Neighbouring facts get a scaled copy of the same shift regardless of whether that shift is correct for them, which is why ripple failures produce wrong answers rather than merely stale ones.

2. Efficacy is a measurement of the optimiser. Near-perfect efficacy tells you the constrained solve worked. Generalisation, portability, locality and robustness are the axes that predict what a user sees, and they are the axes that degrade.

3. Batch and sequential results are not comparable. MEMIT's 10,000-edit numbers come from a single-pass solve. Deployment applies corrections one at a time over months, and that regime enters catastrophic forgetting around 1,400 edits for the same method.

4. The localisation story and the editing method came apart. Causal tracing motivated locate-then-edit and does not predict where editing succeeds. The technique works for reasons other than the ones it was proposed for, which is a standing reason to distrust extrapolation from it.

5. Locality on random negatives is close to free. Overlap between a randomly drawn key and the edit direction is near zero by construction in high dimensions. Any locality claim that does not use near neighbours as negatives is uninformative.

6. Part of the collapse literature was a bug, and the limit survived the fix. r-ROME removed the disabling-edit pathology and moved the cliff; WikiBigEdit then found the cliff again, further out, on realistic data. Distinguish implementation artefacts from scaling limits before dismissing or accepting either.

7. Context beats weights for anything that must compose. A retrieved fact is visible to every circuit in the forward pass, so it participates in multi-hop reasoning the model already does. A written fact is visible only where its key is matched.

8. Weight edits have no rollback without an edit log. Edits compose destructively. If you cannot replay them from the base checkpoint, your only undo is discarding every correction you have made.

Open Questions

Can null-space methods be made to compose? AlphaEdit's projection is measured to improve locate-then-edit methods by roughly 36% on average, but the null space is finite-dimensional and it is an open question whether a principled schedule for spending it exists, or whether the technique simply postpones the same collapse.

Is multi-hop composition over retrieved facts a retrieval problem or a reasoning problem? WikiBigEdit measured retrieval leading on every axis except multi-hop. Whether better retrieval, better prompting or a genuinely different reasoning mechanism closes that gap is not settled.

What is the right unit of correction? Editing literature assumes the subject-relation-object triple. Real corrections are often events with a time interval, which is why continuous-drift benchmarks find both RAG and learning-based adaptation wanting on temporal consistency. A representation that carries validity intervals is plausibly a better target than a triple, and no editing method currently takes one.

Does editing behave differently in reasoning models? Nearly all published results are on base or lightly instruction-tuned models. How an edit interacts with a long chain of thought, which may re-derive the original fact from other associations mid-trace, is largely unmeasured.

Is there a regime where parametric updates are simply required? Safety-motivated removal is the clearest candidate, since the requirement is about what the weights can produce, not about what the context says. Whether editing is a sound tool for that, or whether it only hides the association behind a different key, is an open and consequential question.

Sources and Further Reading

1. Geva, M., Schuster, R., Berant, J., & Levy, O. (2021). "Transformer Feed-Forward Layers Are Key-Value Memories." EMNLP 2021. aclanthology.org/2021.emnlp-main.446
2. Mitchell, E., et al. (2022). "Fast Model Editing at Scale." ICLR 2022. arXiv:2110.11309
3. Mitchell, E., et al. (2022). "Memory-Based Model Editing at Scale." ICML 2022. proceedings.mlr.press/v162/mitchell22a
4. Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). "Locating and Editing Factual Associations in GPT." NeurIPS 2022. arXiv:2202.05262
5. Meng, K., Sen Sharma, A., Andonian, A., Belinkov, Y., & Bau, D. (2023). "Mass-Editing Memory in a Transformer." ICLR 2023. arXiv:2210.07229
6. Hase, P., Bansal, M., et al. (2023). "Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models." arXiv:2301.04213
7. Zhong, Z., Wu, Z., Manning, C. D., Potts, C., & Chen, D. (2023). "MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions." EMNLP 2023, 15686-15702. aclanthology.org/2023.emnlp-main.971
8. Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., & Ghassemi, M. (2023). "Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors." arXiv:2211.11031
9. Cohen, R., Biran, E., Yoran, O., Globerson, A., & Geva, M. (2024). "Evaluating the Ripple Effects of Knowledge Editing in Language Models." TACL, 12, 283-298. aclanthology.org/2024.tacl-1.16
10. Gupta, A., Rao, A., & Anumanchipalli, G. (2024). "Model Editing at Scale leads to Gradual and Catastrophic Forgetting." Findings of ACL 2024. arXiv:2401.07453
11. Gupta, A., Baskaran, S., & Anumanchipalli, G. (2024). "Rebuilding ROME: Resolving Model Collapse during Sequential Model Editing." arXiv:2403.07175
12. Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., & Chen, H. (2024). "WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models." arXiv:2405.14768
13. Fang, J., Jiang, H., Wang, K., Ma, Y., Shi, J., Wang, X., He, X., & Chua, T.-S. (2025). "AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models." ICLR 2025 (outstanding paper). arXiv:2410.02355
14. Huang, B., Chen, C., Xu, X., Payani, A., & Shu, K. (2025). "Can Knowledge Editing Really Correct Hallucinations?" ICLR 2025. arXiv:2410.16251
15. Thede, L., Roth, K., Bethge, M., Akata, Z., & Hartvigsen, T. (2025). "WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs." ICML 2025. arXiv:2503.05683
16. "RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World." Findings of ACL 2026. aclanthology.org/2026.findings-acl.546

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.