Evaluating Knowledge Edits

An editing method that scores 99 percent efficacy has established that, when given the exact prompt it was tuned on, the model emits the token it was tuned to emit. That is a check on the optimiser, not on the model's knowledge. Almost every disappointment with knowledge editing in production traces back to accepting that number as the result.

The five axes

Current benchmarks decompose edit quality into efficacy, generalisation, portability, locality and robustness. HalluEditBench evaluates all five over a dataset of more than 6,000 real hallucinations spanning nine domains and 26 topics, and reports that performance beyond efficacy is generally unsatisfactory across methods (Huang et al., 2025, Can Knowledge Editing Really Correct Hallucinations?, ICLR 2025, arXiv:2410.16251).

Robustness is the axis newest to the list and the one closest to a security property: does the edit survive a rephrasing, a system prompt, a multi-turn conversation, or a user who argues with the model?

The pre-edit condition nobody checks

HalluEditBench's other contribution is methodological. Standard editing datasets do not verify that the model was wrong before the edit. If the model already knew the answer, a successful "edit" measures nothing, and a benchmark full of such cases inflates every method equally. Constructing an evaluation set from hallucinations the target model actually produces is more work and is the only way the efficacy number means what it appears to mean.

The same logic applies in a production setting. Before editing anything, record the model's current answer, the prompt that produced it, and the answers to a handful of neighbouring questions. That record is both the pre-condition and the regression test.

Locality needs a hard negative set

Locality is usually measured on random unrelated prompts, which is too easy: a rank-one edit rarely disturbs a randomly chosen fact. The informative set is the near neighbours, other facts about the same subject, the same relation applied to similar subjects, the same object reached by a different relation. Those are the keys with real overlap with the edited direction, and the ones where collateral change actually lands.

Scale changes the ranking

Method rankings measured on a few hundred edits do not survive contact with realistic volume. On WikiBigEdit, which draws over 500,000 question-answer pairs from real Wikidata changes, locate-then-edit methods collapse within the first 10,000 updates, methods built for high volume converge back toward pre-edit accuracy, and a straightforwardly implemented retrieval-augmented baseline leads on every axis except multi-hop reasoning, holding around 78.6 percent on rephrasings and 66.2 percent under persona conditioning (Thede et al., 2025, WikiBigEdit, ICML 2025, arXiv:2503.05683).

An evaluation that stops at 100 edits will recommend the wrong method for a system that will accumulate 100,000.

When it breaks

Editing score conflates axes. Composite metrics such as the CounterFact editing score average efficacy, generalisation and locality into one number, so a method can climb by winning the easy axis. Report the components.

Single-model results do not transfer. Edit behaviour differs across architectures and sizes; a method tuned on GPT-2 XL may behave differently on a 70B instruction-tuned model with a different MLP width and normalisation.

Downstream ability is a separate test. Factual metrics will not reveal that instruction-following or arithmetic degraded. Keep a general benchmark in the harness and run it after every batch of edits.

Batch and sequential protocols are different experiments. Say which one you ran. The numbers are not comparable, and the sequential case is the one that matches how a deployed system accumulates corrections.