?
Problem

BioMysteryBench’s v11 Audit Revised Tasks, Answer Keys and Scoring

An answer key is the reference used to judge a response. Anthropic’s v11 changelog (version dated July 6, 2026) reports a June audit with expert reruns, repeated model attempts and checks against underlying data. Nine removals—including wrong keys, missing information and questions lacking one recoverable answer—reduced 99 tasks to 90; 24 other tasks received 28 edits. For every task, scoring now explicitly required a correct answer and no cheating, not merely passing an anti-cheating check. Wrong keys or unavailable information can therefore mimic solver failure. This is a developer audit, not independent replication; no comparable model rerun rate for the corrected 90 tasks is supplied.

0

1

?
Updated 2026-11-09

Contributors are:

Who are from:

Tags

Scientific Oversight of AI-Assisted Research

1Cademy