Google DeepMind, alongside the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, has launched a pilot demonstrating double-blind evaluations for proprietary frontier artificial intelligence models. The initiative evaluates Gemini Flash Lite inside hardware-isolated secure enclaves to resolve the structural conflict between model intellectual property and benchmark confidentiality.
External evaluations of commercial large language models traditionally face a mutual trust barrier. Independent auditors and safety organizations cannot inspect raw model weights or proprietary inference stacks without risking the developer's commercial secrets. Conversely, model developers cannot run unreleased, high-stakes benchmark test suites on their own infrastructure without risking benchmark contamination, where test questions inadvertently enter future pre-training or fine-tuning datasets.
Hardware Enclaves and Structured Transparency
The pilot framework, documented in DeepMind's technical report "Resolving the Dual Confidentiality Dilemma in AI Safety Auditing," deploys a hardware-enforced Double-Blind Evaluation (DBE) protocol built on Google Cloud Confidential Space.
The DBE architecture isolates both the evaluation data and the model weights inside a temporary trusted execution environment (TEE):
- Memory-level hardware encryption: Execution takes place inside AMD SEV-SNP secure enclaves, encrypting memory at runtime to prevent the cloud host, Google engineers, and external auditors from inspecting execution states.
- PySyft Datasite orchestration: OpenMined's PySyft system manages structured transparency policies, establishing cryptographic identity and governing how data enters the enclave.
- Cryptographic attestation: Before either party releases encrypted assets to the enclave, the system generates hardware-signed cryptographic measurements verifying that only approved, unmonitored evaluation code is running.
- Zero-leakage execution: Inside the enclave, Gemini Flash Lite processes the confidential benchmark prompts. The evaluation harness computes aggregate performance and safety metrics, outputs the final score report, and tears down the environment without writing prompt logs or raw intermediate tokens to persistent storage.
Preventing Benchmark Contamination in Safety Audits
Benchmark contamination has undermined public model leaderboards as frontier labs scale web scrapes for pre-training corpora. When models inadvertently ingest static benchmark datasets, reported evaluation scores measure training memorization rather than reasoning generalization.
This issue is acute for high-stakes audits conducted by government safety institutes, including evaluations for autonomous cyber-offense, biological risks, and automated exploitation. Keeping evaluation suites strictly confidential has previously required either trusted third-party escrow or blind remote API queries. Remote API testing remains vulnerable to provider-side prompt logging, caching, and post-hoc data retention.
By proving cryptographic isolation during model execution, the DBE architecture establishes a path for regulators and independent evaluation bodies like MLCommons and the Singapore AISI to run unreleased tests against commercial models without proprietary exposure on either side.
