Ripple Effects and Edit Portability

Tell a model that Jack Depp's father is Johnny Depp and you have not inserted one fact, you have implied many: Jack is Lily-Rose's sibling, Johnny has a son, Jack's grandparents are Johnny's parents. A human who accepts the first statement accepts the rest. An edited model, in general, does not.

This is the ripple effect, and it is the clearest single demonstration that weight editing changes a lookup rather than a belief (Cohen et al., 2024, Evaluating the Ripple Effects of Knowledge Editing in Language Models, TACL 12:283-298).

What the benchmarks measure

RippleEdits is a diagnostic set of 5,000 factual edits evaluated along six criteria: logical generalisation, two forms of compositionality, subject aliasing, preservation of unrelated facts, and relation specificity. Prominent editing methods fail to make consistent changes across these criteria. The result that stung is that a simple in-context editing baseline, which just puts the new fact in the prompt, scores best on the benchmark.

MQuAKE attacks the same weakness from the reasoning side, with multi-hop questions whose answers must change as an entailed consequence of the edit. Editing methods recall the edited fact accurately and then fail on the multi-hop questions, with accuracy falling as hop count rises from two to three to four, and falling further when several edits interact (Zhong et al., 2023, MQuAKE, EMNLP 2023, arXiv:2305.14795). The paper's own remedy, MeLLo, stores edited facts externally and prompts the model iteratively, which is again a retrieval design rather than a weight edit.

Why entailment does not propagate

A rank-one edit rewrites the mapping from one key direction to one value direction. The consequences of a fact are not stored adjacent to it: "sibling of" is computed by a different circuit reading different keys, and nothing in the edit objective touches that circuit. There is no mechanism by which changing the parent association updates the sibling association, because the model never derived the second from the first in the first place; both were learned independently from data that happened to be consistent.

This is why in-context editing wins on ripple benchmarks. A fact in the prompt is available to every circuit that reads the residual stream, including the ones that do multi-hop composition at inference time. A fact in the weights is available only where the edited key is matched.

Portability as a distinct metric

The field now separates four properties, and a method can be excellent at the first and useless at the rest:

• Efficacy: the edited prompt produces the new answer.
• Generalisation: paraphrases and aliases of the edited prompt also produce it.
• Portability: downstream questions whose answers depend on the edit change accordingly.
• Locality (specificity): everything else stays as it was.

Headline numbers in papers are usually efficacy, sometimes generalisation. Portability is where the gap is widest, and it is the property that determines whether an edit is useful to a user who does not phrase questions the way the benchmark does.

When it breaks

Aliases are cheap failures. An edit made on "Eiffel Tower" may not fire on "la Tour Eiffel" or "the tower in Paris designed by Gustave Eiffel", even though a paraphrase is the easiest ripple to demand.

Conflicting neighbours. An edit can create incoherence: the model now reports a new employer while continuing to report the old colleagues, office and job title that the old employer entailed. A confidently inconsistent model can be worse for users than an outdated one.

Reverse relations rarely follow. Editing "A is the capital of B" does not reliably update "B's capital is A", because the two directions use different keys.

Benchmarks disagree with each other. Scores on CounterFact, zsRE, RippleEdits and MQuAKE are not interchangeable, and a method tuned to one can look much worse on another. Evaluate on the shape of change your application actually makes.