By William Isaac, Sol Messing and Kristian Lum

Google DeepMind has introduced the world's first double-blind evaluation of a proprietary, frontier-class AI model, an approach that keeps external evaluations confined to a cryptographic "box" where they cannot be used by models later to optimize performance ahead of testing.

The initiative aims to build trust in proprietary model benchmarks using cryptographically secure environments. Consider a student set to take a high-stakes exam: if the student accidentally peeks at the test questions in advance, a perfect score is influenced by that knowledge and becomes a meaningless accomplishment. To truly measure what they know, the student must have no visibility of the test questions until it is time to sit the exam.

That is the exact challenge the industry faces when evaluating advanced AI models. If a model has already seen the test questions - a problem known as benchmark contamination - the results can only be trusted to an extent. The concern is well documented in the research literature, where studies have found public benchmark test sets leaking into the web-scale corpora used to train large models, prompting dataset maintainers to add tools such as canary strings that let developers detect whether a benchmark appeared in training data.

The pilot is being conducted in partnership with the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons. Together, the partners will test a Gemini Flash Lite model against confidential benchmarks in a privacy-preserving environment, increasing evaluation integrity.

Google states that it assesses its AI systems using a broad spectrum of evaluations throughout model development and deployment, but does not rely on internal testing alone. To identify potential blindspots, the company works with a diverse group of external partners - including specialized research labs, civil society and national AI Safety and Security Institutes (AISIs) - using their unique expertise to stress-test its models. AISIs are themselves a recent institutional development: in late 2024, AI safety institutes from governments including the United States, the United Kingdom, Japan, Singapore, South Korea and the European Union formed an international network to coordinate on evaluating advanced AI systems.

As AI models become more capable, ensuring a model has not seen the test questions or prompts in advance is critical, since prior exposure can skew results. Policymakers, researchers and enterprises need to trust that AI benchmarks accurately reflect a model's true capabilities and safety, but if models are able to "peek" at evaluation questions in advance, scores can be artificially inflated and that trust undermined.

Although zero-logging protocols and rigorous contractual safeguards have long kept external test prompts confidential, incorporating technical and cryptographic safeguards marks a major step forward in secure model evaluation.

How double-blind evaluations work

Historically, high-stakes external evaluations required a tradeoff. Either evaluators handed over their testing prompts, risking the model provider seeing the test questions in advance, or the model provider handed over their model weights, risking their intellectual property.

Double-blind evaluations eliminate this compromise. By using Confidential Space within Google Cloud's Confidential Computing portfolio, both parties can cryptographically verify that the external evaluation data and the proprietary model remain private to their respective owners. The evaluator cannot see the Gemini model weights, and Google cannot see the evaluator's test prompts. Confidential computing itself is an established technology: it relies on hardware-based trusted execution environments that keep data encrypted even while it is being processed, an approach already used in regulated sectors such as healthcare and finance.

A novel approach to building trust in model evaluations

This cryptographic evidence helps prevent benchmark contamination and protects sensitive data. As models become more capable, this becomes particularly important for highly sensitive evaluations, such as those used for cybersecurity or by government bodies. Double-blind evaluations unlock the ability for independent organizations to rigorously test advanced models without compromising data sovereignty or security.

Google hopes this pilot establishes a new frontier for model oversight, helping the broader industry build safer, more reliable and widely trusted AI systems. Further detail on the methodology and findings is available in the technical report.