Google DeepMind has introduced a framework for conducting double-blind evaluations of proprietary frontier AI models within cryptographically isolated computing environments. The initiative, developed in partnership with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, aims to resolve the tension between protecting benchmark datasets from contamination and safeguarding proprietary model weights.
In traditional third-party model evaluations, organizations face an unavoidable compromise. Evaluators must either provide their proprietary test prompts directly to the model creator, creating risks of data leakage and training set contamination, or model developers must distribute raw model weights to external auditors, risking intellectual property exposure.
DeepMind's double-blind evaluation architecture relies on Confidential Space within Google Cloud Confidential Computing. By executing the model and the evaluation harness inside hardware-attested GPU enclaves, the system guarantees that neither party can inspect the other's assets during execution.
Cryptographic Enclaves and Zero-Knowledge Benchmarks
The pilot deployed Google's Gemini Flash Lite model against confidential benchmark suites. Under the isolated workflow:
- Evaluators package and encrypt their test suites, maintaining full control over benchmark queries and scoring rubrics.
- The model provider deploys the model into a secure, hardware-isolated enclave without releasing unencrypted weights externally.
- Enclave execution produces verifiable cryptographic proofs confirming the exact code executed and the integrity of the evaluation run.
- Model outputs are scored and returned to the evaluator without logging prompts to persistent training corpora or developer-accessible telemetry.
This process prevents benchmark contamination, a recurring challenge where models inadvertently memorize public or shared test questions, artificially inflating benchmark scores without corresponding gains in general reasoning capability.
Industry-Wide Implications for Safety and Auditing
Standardizing double-blind evaluations allows national safety institutes and independent auditors to test frontier AI systems against sensitive red-teaming benchmarks, including cybersecurity vulnerability discovery and national security evaluations, without exposing sensitive attack vectors or proprietary architectures.
MLCommons and the participating safety institutes plan to build on this pilot to establish standardized protocols for privacy-preserving AI audits across commercial and open-weight models.
