Google DeepMind's double-blind AI evaluation pilot

Google DeepMind, Singapore's AI Safety Institute, OpenMined, AVERI, and MLCommons completed a pilot double-blind evaluation of Gemini 2.5 Flash-Lite using Google Cloud's Confidential Space. The evaluation is the first of its kind for a proprietary frontier model: hardware attestation verifies exactly what code runs before either party's assets enter the enclave, so the evaluator cannot see Gemini model weights and Google cannot see the evaluator's test prompts. AVERI encrypted evaluation prompts with a private key invisible to all other parties, jointly ran the evaluation inside a Google-configured enclave using OpenMined software, and alone decrypted and graded outputs.

The pilot targets benchmark contamination: when a model trains on the same questions used to grade it, its scores measure memorization rather than capability. Previously, high-stakes external evaluations required a tradeoff between exposing test prompts to model developers or exposing model weights to evaluators; the double-blind setup eliminates that compromise. A 2024 pilot by OpenMined, Anthropic, and the AI Security Institute previously demonstrated the secure enclave mechanism using GPT-2 and a five-row evaluation; this pilot advances to a production model and a more comprehensive evaluation.