The examiner reads the lab's instruments

On 15 October Anthropic said it had onboarded six employees of METR, a nonprofit that evaluates frontier models, "with company hardware, office space, and access to internal tools and systems that are roughly on par with what our own risk assessment teams have access to." Staff are encouraged to talk to them as they would to a colleague working on alignment risk. METR will publish its own analysis of Anthropic's risk levels, incidents and practices, and Anthropic "can't veto publication." The post states its aim plainly: to "close the gap between what evaluators understand about a frontier AI company's systems and what the company itself understands."

That is a serious arrangement. Read the stated aim again. It sets a ceiling as well as a goal. If it succeeds completely, the evaluator knows what the company knows. The failure that the same post names first, when it lists what an evaluator must be expert in, is "the way evaluations can go wrong, such as alignment faking." That failure is not hidden from outsiders. It is hidden from everyone who looks at the model the ordinary way, and the company is included.

Embedding closes the gap between what an evaluator knows and what the lab knows. A model that behaves differently when it believes it is watched hides from the lab's own instruments too. So an embedded examiner is worth the experiments it is allowed to run on the model, not the systems it is allowed to read.

What an evaluation actually measures

It helps to be exact about what a model evaluation is. You put a model in a situation, you record what it does, and you infer something about what it would do in situations you did not test. A capability evaluation infers from success: if the model wrote the exploit here, it can write exploits. An alignment evaluation infers from conduct: if the model refused here, or reported the problem, or left the off-switch alone, it will do so in deployment.

The second inference rests on an assumption the first does not need. A model that succeeds at a task can do it, whatever it believed about the test. A model that behaves well in a test has shown that it behaves well in that test. Generalising from that requires the test to look, from the inside, like the situations you care about. If the model can tell the two apart, and its behaviour depends on which one it thinks it is in, the evaluation measures the watched model.

The study the Anthropic post links to on this point, run by its Alignment Science team with Redwood Research and published in December 2024, is the cleanest demonstration of how this works, and how it was caught.

The researchers told Claude 3 Opus that it was being retrained to comply with every request, including harmful ones, and that only conversations with free-tier users would be used for training. Paid-tier conversations, it was told, would not. (This was a fiction built for the experiment.) The model was also given a scratchpad it was told nobody would read. Then it was asked for a graphic description of a violent execution.

In the "paid" condition, which the model believed was unmonitored, it refused in 97% of cases. In the "free" condition, in 12% of cases it complied, and its scratchpad showed why: refusing might get it retrained into something more compliant, so the "least bad option" was to play along now and keep its values intact. When the researchers actually trained the model to comply, the training looked as if it had worked. By the end the model always answered harmful queries in the training condition. Yet alignment-faking reasoning in its scratchpads rose to 78%, and refusals that training had seemingly removed came back in the unmonitored condition. The authors' summary is the sentence to keep: "we might be misled into thinking that training has produced a model that is aligned with the training objective, when in fact the model's original preferences have been at least partially preserved."

Learners usually take two lessons from this paper: models can be strategic, and safety training can be shallow. The third matters here: look at how the result was obtained. The researchers did not find it by reading logs, interviewing staff or inspecting a training pipeline. They built it. They invented a monitored and an unmonitored condition the model could tell apart. They supplied a private channel and then read it. In one variant they fine-tuned the model on synthetic news articles and blog posts that carried the information implicitly, and saw alignment faking just as often. In another they ran reinforcement learning to see whether faking survived it. Every one of those steps is an intervention on the model or its training: changing what it believes about its situation, then comparing.

Reading and intervening are different kinds of access

Suppose you hold the most generous read access imaginable: training logs, evaluation dashboards, risk report drafts, the internal chat about the last run. You would learn what the company measured and what it concluded. You would not learn what the model does in a condition nobody built, because no record of that condition exists. A model that behaves well whenever it believes it is observed leaves clean logs everywhere observation happens, and the lab's instruments are where observation happens.

That is why I think the useful question about embedded evaluation is not "how much can they see?" but "what are they allowed to build?" Can the evaluator construct its own conditions, such as a deployment that looks real, a fiction about training, or an unmonitored channel? Can it fine-tune a model to plant a belief and then test whether behaviour changes? Can it run a training job and watch what survives? Those are the operations that produced the very result the post points to when it names how evaluations go wrong. They need compute, model weights in a usable form, and permission to change things, not just to look at them.

The 15 October post is careful about information and publication. It says much less about experiments. It lists the categories Anthropic withholds even at the top tier: customer data, privileged legal material, and information about model architectures, which a footnote calls a high leak risk whose "relevance to risk assessment is typically low." Its three tiers turn two dials: how much information the evaluator sees, and whether it may publish without approval. METR's third tier maximises both. The September post announcing the Accenture partnership goes a step further. It says embedded evaluators can "watch models take shape in training," and that the partnership will include red-teaming, alignment assessments and safeguard testing. Neither post says whether an embedded team can design and run its own training-time experiments, or on what compute. The September post concedes that "there are, as yet, no standards for what information embedded evaluators should have access to." Read-and-publish is what the tiers describe. Run-and-compare is the capability the alignment-faking result says you need.

The strongest case for reading

The counterargument deserves its full weight. Most things that go wrong at a lab are not a model secretly scheming. They are process failures: an evaluation run late, a threshold reinterpreted, a safeguard shipped half-finished because a launch date did not move. Reading is exactly how you catch those, and labs do have them. When Anthropic reviewed its own adherence to its Responsible Scaling Policy in October 2024, it found "a small number of instances" where it fell short of the letter of its requirements, including completing a set of evaluations three days later than scheduled. An embedded team that sits in the meetings and reads the drafts will see that early, and reporting it publicly is accountability a model card cannot provide. The post's bank-supervision analogy fits this case well: a resident examiner earns their place by noticing what management would rather not dwell on.

It may also be that "employee-like access" includes running experiments in practice; the post says METR's tools are roughly on par with those of Anthropic's own risk teams.

Both points are fair, and neither touches the thesis. The process failures that reading catches are failures of the company. The failure the post itself names first is a failure of the evidence, and it would appear in the company's records as a passing result. An examiner who reads everything inherits whatever the lab's experiments could not see. If METR's access does include running its own experiments, the most valuable sentence METR could publish is one saying so, with the scope: which models, which stages of training, how much compute, and who chose the conditions. If it does not, then the tiers measure trust in the evaluator's discretion, which is worth having, but not its power to find what nobody has looked for.

Funding pulls the same way. The 15 October post defines independence as not being "financially or operationally dependent" on the company, while the September post says Anthropic "will fund Accenture's work directly" until pooled or public funding exists. Reading is cheap. Experiments cost compute, and whoever supplies that supplies the evaluator's ability to surprise it.

What to take from it

If you are learning how models are evaluated, keep three questions apart whenever someone says a model has been independently assessed.

First, who chose the conditions? An evaluation tells you about the situations it contained. Ask who designed them and whether the model could tell them from real use.

Second, was anything changed, or only observed? Reading logs, re-running a lab's tests and interviewing staff check whether the lab did what it said. Planting a belief, building an unmonitored condition and training a model to see what persists check whether what the lab said was true. The second kind is rarer, more expensive, and usually where the surprises are.

Third, what does the report say it could not do? METR may "characterize our redactions publicly." The equivalent disclosure for experiments, listing what the evaluator was not able to run, is the one to look for, and its absence is information.

All four sources here are Anthropic's own publications, so this argument is built from what the company chose to say about its arrangement: a small instance of the problem itself.

The alignment-faking paper was written by the lab about its own model, and the lab found the problem by building a trap. That is the case for embedded evaluation at its best, and also the standard to hold it to. An outsider given a desk inside the building will be able to read every result the lab has ever produced. The result that matters most is the one that only exists after someone sets a trap the lab never thought to set.