Google DeepMind, in collaboration with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, has piloted a cryptographic framework for double-blind evaluations of proprietary frontier language models. The pilot, conducted on Gemini 2.5 Flash Lite, uses hardware-isolated confidential computing to ensure that model developers cannot see evaluation prompts while evaluators cannot inspect proprietary weights or inference code.
The project addresses benchmark contamination and intellectual property exposure, two persistent obstacles in independent AI safety auditing.
The Dual-Confidentiality Dilemma
Independent evaluation of frontier artificial intelligence models has historically required difficult trade-offs between test integrity and proprietary security.
When external auditors, academic groups, or government safety institutes evaluate a commercial model, two competing confidentiality requirements emerge:
- Benchmark Secrecy: Evaluators must keep their test suites, canary tokens, and sensitive evaluation prompts confidential. If test datasets leak or are logged during inference, model developers can inadvertently or intentionally train on the data, rendering subsequent benchmark scores invalid.
- Model Intellectual Property: AI developers must protect proprietary model weights, runtime configurations, and inference code from unauthorized extraction, reverse engineering, or weight theft.
Prior safety evaluations have relied primarily on legal non-disclosure agreements, zero-data-retention API agreements, or trusted third-party auditing environments. However, contractual assurances do not provide mathematical or hardware-level proof against data exposure, runtime logging, or memory inspection.
Hardware-Enforced Isolation Architecture
To replace administrative trust with cryptographic guarantees, DeepMind implemented an evaluation pipeline running within Google Cloud Confidential Space.
The technical architecture relies on three primary hardware and cryptographic components:
- Confidential Virtual Machines: The execution environment runs inside an isolated compute instance with memory encrypted via Intel Trust Domain Extensions (Intel TDX) and NVIDIA H100 Confidential GPUs. Host operating systems, hypervisors, and cloud infrastructure administrators cannot read memory contents during computation.
- Remote Attestation and Identity Tokens: Before any confidential assets are loaded, the runtime environment generates a cryptographically signed hardware attestation token. Google Key Management Service verifies this token to confirm that the enclave runs untampered code images.
- Multi-Party Key Release: Once attestation passes, two distinct decryption keys are released into the enclave: one from Google to decrypt the model weights and JAX C++ inference runtime, and one from the external evaluators (managed via OpenMined PySyft tooling) to decrypt the test datasets and scoring scripts.
During evaluation, model weights are streamed into the enclave over local Unix Domain Sockets. Inference requests execute purely within the encrypted memory boundaries. Google engineers cannot observe the incoming evaluation prompts, and the external evaluators cannot inspect or download the underlying model parameters.
Pilot Deployment and Results
For the initial validation pilot, DeepMind evaluated Gemini 2.5 Flash Lite against confidential test suites provided by MLCommons and the Singapore AI Safety Institute.
The evaluation framework verified:
- End-to-End Encryption: Datasets and model weights remained encrypted at rest, in transit, and during in-memory processing.
- Zero-Leakage Output Routing: The enclave generated and cryptographically signed the final benchmark results, transmitting only the aggregated scores back to the external evaluators without exposing intermediate token activations or raw prompts to either party.
- Reproducible Performance: Benchmark execution inside the confidential enclave completed with negligible performance and latency overhead compared to standard inference deployments.
Path to Standardized AI Auditing
The double-blind methodology is intended as a repeatable standard for third-party auditing, regulatory compliance, and sovereign safety institutes. In high-consequence domains such as biosecurity, autonomous cyber operations, and critical infrastructure resilience, evaluation suites often contain sensitive vulnerability data that cannot be shared with private vendors.
By enforcing isolation at the silicon level, independent auditors can execute rigorous evaluations against proprietary frontier models without requiring model providers to surrender weight custody or risking benchmark contamination.
