Anthropic has placed a team from METR, the nonprofit model evaluation group, inside the company with what it calls "employee-like access" to its systems and staff. The arrangement, announced on October 15 and reported in more detail by the Financial Times via Ars Technica on October 18, is the first concrete version of the embedded auditor idea that frontier labs have been discussing for months.
The core terms are simple. METR gets close to the same view of Anthropic's risk picture that Anthropic's own risk teams have, and METR can publish what it finds without Anthropic's sign-off.
What METR actually gets
According to Anthropic, it has onboarded six METR employees and given them company hardware, office space, and access to internal tools and systems "roughly on par" with what its own risk assessment teams use. Anthropic staff are encouraged to share information with the METR team as they would with a colleague working on alignment risk.
There are carve-outs. Anthropic lists a small set of predefined categories that stay off limits or can be redacted:
- data about customers
- attorney-client privileged information
- information about model architectures
On architecture, Anthropic says the leak risk is high and the relevance to risk assessment is usually low. When architecture details are highly relevant, it says it will generally seek to share them, with explicit internal approval.
The embedded team works under enforceable agreements, and other METR staff, including researchers who work with Anthropic on separate projects, are walled off from its work.
Publication rights, with a catch
METR will publish its analysis of Anthropic's risk levels, incidents, or practices. Anthropic says it cannot veto publication. It can require redactions if material falls into the predefined categories above.
The catch is that Anthropic decides what falls into those categories. METR keeps the right to make statements about the level of access and information it received, which gives it a way to flag if it was kept from something. That is a meaningful check, but it is not the same as full independence over what gets disclosed.
The FT report adds two details not in Anthropic's post: Anthropic is not paying METR for the arrangement, and METR does not accept donations from AI companies or their staff. METR is funded by individuals, philanthropic organizations, and government bodies including the UK AI Security Institute, according to the report.
A three-tier model for outside evaluators
The more durable part of the announcement may be the framework Anthropic says it is building for all external evaluators. It describes three tiers:
- Tier 1: narrowly scoped risk investigations. Anthropic decides what access to provide, and findings are published only with its approval.
- Tier 2: more information, and the evaluator can publish without Anthropic's approval. Anthropic can still require removal of specific categories of information, but the evaluator can publish its view on whether those redactions were justified.
- Tier 3: the METR arrangement, with employee-like access and tier 2 publication rights.
Anthropic says it hopes to eventually have several evaluators at tier 3, each publishing its own analysis of the company's risk posture and key decisions.
Other partners sit lower on that ladder for now. SecureBio is reviewing the chemical and biological weapons chapter of Anthropic's August Risk Report, after publishing a pilot review of the previous report. Anthropic says it is working with Accenture and Faculty on evaluation, red-teaming, alignment assessments, and safeguards testing, and is in early discussions with other groups including Hugging Face.
The bank examiner analogy
Anthropic frames the setup with a comparison to bank supervision. Large banks know more about their own risk positions than outsiders do, so regulators place resident examiners inside them. Anthropic argues the same logic applies to frontier AI: some safety-relevant information cannot be published because it could help malicious actors or is commercially sensitive, so the next best option is to give trusted outsiders deep access and let them report conclusions.
The analogy has limits that Anthropic acknowledges. Bank examiners work for regulators with legal authority. METR is a nonprofit operating under a voluntary contract with the company it is evaluating. The FT notes that critics of the embedded evaluator approach have raised conflict-of-interest concerns, pointing to problems in the accounting sector, and argue that government regulation is what is actually needed.
Why it matters
Most third-party AI evaluation to date has been pre-release model testing: an outside group gets API access to a model for a few weeks and writes up what it found. That tells you about a model, not about the company's training process, internal incidents, or how its safeguards work in practice. Embedding evaluators with internal access is an attempt to cover that gap.
It also sets a reference point for competitors. Anthropic says it signed the White House Accord on Super Intelligence, under which companies pledged to work with independent external auditors or evaluators, and closes its post by saying every frontier AI company should embed evaluators. Per the FT, OpenAI has not yet laid out its own plan.
Whether this becomes a real oversight mechanism depends on what METR publishes and whether it ever uses its right to say it was denied access. The first published METR assessment under this arrangement will be the real test.
