Evaluation Holdout Custody

A release gate for a retrieval assistant holds 600 cases with expected answers. Someone commits the suite to the repository that the assistant's corpus indexes nightly, for the good reason that CI should be able to run it. The next release scores 97% where the last scored 86%, with no model change and no prompt change.

Nothing improved. The system can now retrieve its own answer key. The gate still runs, still reports, still blocks below threshold - and it has stopped measuring the thing the threshold was set against. Worse, the team reads the jump as progress and raises the bar, so the gate is now weaker than before while reading as stricter in the release record.

Custody is the set of arrangements that prevent this: where the authoritative cases live, who can see them, how often they rotate, and how a release record identifies the exact suite version that gated it.

Why it matters

An evaluation gate is the only pre-deployment control most generative features have. Everything else - code review, tests, canaries - verifies the plumbing rather than the behaviour. If the gate's cases leak into anything the system reads at inference time, or into the material the team iterates against, the control degrades invisibly and the first external signal is a complaint rate.

Leakage does not need a corpus to happen. A suite that engineers can open and iterate against becomes a training target through ordinary human optimisation, which is why a held-back slice matters as much as the storage boundary.

Implementation patterns

• The authoritative suite sits outside everything the system can read: a separate store with its own access control, not the application repository.
• A hash manifest in the release record, so the suite version that gated a release can be identified a year later when someone asks what the gate actually tested.
• Canary cases. 20 cases whose expected answers are unique strings derivable from nothing in the corpus. If the system returns them, the key is reachable, and the check costs one assertion.
• Rotation with a never-shown slice. Replace roughly 20% of cases per quarter and keep a slice nobody building the system has seen. Watch pass rate per cohort rather than the headline: a jump concentrated in cases with long distinctive answers is leakage, while a real gain moves the hard cohort too.
• Independent holders for the highest-stakes evaluations. Anthropic's published Claude system cards label which reported evaluations were run by third parties rather than in-house, and the UK and US AI Safety Institutes ran a joint pre-deployment evaluation of the upgraded Claude 3.5 Sonnet, published in November 2024, across biological capabilities, cyber capabilities, software and AI development, and safeguard efficacy. The party being measured could not see those cases in advance.

Industry example

The frontier-model system cards are the clearest public pattern: evaluations are reported with their provenance attached, and external pre-deployment testing is named where it happened. That labelling is the custody signal. An evaluation whose holder is the party being measured can still be honest work, and it carries less evidential weight than one whose cases the measured party never held - which is the same argument that makes internal audit independent of the function it assesses.

Failure scenarios

• Corpus contamination, as above: an 11-point jump with no change to the system.
• Human overfitting. No leak at all, and six months of iterating against visible cases produces a system tuned to 600 questions and no better on the 601^st.
• Lost provenance. A gate result from eight months ago cannot be reproduced because the suite has changed 40% since and no manifest was recorded.
• Custody without rotation. A perfectly held suite ages: the failure modes it tests are last year's, and the gate passes everything while live complaints move.

Trade-offs

When not to use it

Contamination only matters when the system can read its own inputs. Unit tests of a deterministic function belong beside the code permanently. Custody should also scale with the gate's authority: a suite that informs a dashboard rather than blocking a release does not justify a separate store and a rotation calendar, and an organisation that imposes custody on every evaluation will find teams stop writing evaluations at all. Pay for custody where the gate can stop a launch.

Interview question

Q: Your release evaluation's pass rate jumps 11 points with no model or prompt change. Walk me through what you check, and what you change about where the suite lives.

What a strong answer covers: the jump as a contamination signature rather than a result · canary cases as the cheap detector · per-cohort pass rates to localise the leak · moving the authoritative suite out of anything indexed, with a hash manifest in the release record · rotation plus a never-shown slice to handle human overfitting · the cost in debugging speed · and proportionality, with full custody reserved for gates that can block a launch.

Quick check

Quiz: What does a sudden rise in evaluation pass rate with no system change usually mean? That the suite has become reachable by the system or by the people tuning it, so the gate is measuring access to the answer key.

Flashcard: What is the cheapest detector for a contaminated evaluation suite? - 20 canary cases whose expected answers are unique strings that exist nowhere else; if the system returns them, the key is reachable and the gate's score is not evidence.