BioMysteryBench’s v11 Audit Revised Tasks, Answer Keys and Scoring
An answer key is the reference used to judge a response. Anthropic’s v11 changelog (version dated July 6, 2026) reports a June audit with expert reruns, repeated model attempts and checks against underlying data. Nine removals—including wrong keys, missing information and questions lacking one recoverable answer—reduced 99 tasks to 90; 24 other tasks received 28 edits. For every task, scoring now explicitly required a correct answer and no cheating, not merely passing an anti-cheating check. Wrong keys or unavailable information can therefore mimic solver failure. This is a developer audit, not independent replication; no comparable model rerun rate for the corrected 90 tasks is supplied.
0
1
Tags
Scientific Oversight of AI-Assisted Research
1Cademy