BioMysteryBench’s Human-Difficult Label Is Panel-Specific
A benchmark tests performance on specified tasks. Anthropic’s BioMysteryBench report (first published April 29, 2026; last modified October 9, 2026) retains historical results for 99 biological-data questions. Up to five field experts tried each from scratch: 76 were solved by at least one person; 23 remained unsolved by that panel, labeled human-difficult after initial checks. On those 23, Anthropic reports Claude Mythos Preview’s roughly 30% average per-attempt solve rate, using five attempts per question. This tests answer-finding, not understanding a supplied AI answer; equal time and resources were not established. The page is mutable, not a frozen April snapshot. This rate does not describe the later corrected 90-task version.
0
1
Tags
Scientific Oversight of AI-Assisted Research
1Cademy