Case-based inference (synthesis): these results cannot be added as interchangeable evidence of oversight. Lean acceptance is relative to a stated proposition and premises, whereas METR's uncertain, partly redacted growth estimate describes change in task capability at one lab. Even the two model-focused tests differ: MATH reward models rank candidate solutions; the evaluation measures selected final-answer correctness; circuit tracing investigates internal pathways, incompletely. Consequently, a better MATH selection score cannot fill METR's publicly missing evidence or establish that a written explanation matches an internal pathway; a traced pathway cannot substitute for Lean's proof obligation. Combining these results into one oversight metric requires a justified mapping between those different targets, which these cases do not provide. None directly measures a scientist's understanding of unfamiliar research.

Evidence status: analytical_comparison_of_distinct_measurement_targets.

Four Evaluation Results Answer Different Questions

A capability benchmark summarizes performance on specified tasks in a particular evaluation setup. In its 2026-11-05 assessment of Anthropic during October 2026, METR judged model capability growth on well-specified tasks likely around twice its March 2024–October 2025 rate. METR calls this judgment highly uncertain and warns that benchmark growth may overstate real-world acceleration. Anthropic redacted much evidence METR found most informative, limiting public re-examination. This is a one-laboratory capability-growth assessment, not a causal estimate that AI doubled general research productivity. This reported capability-growth result does not directly measure scientists' understanding.

Evidence status: independent_assessment_with_high_uncertainty.

Google

Independent Assessment of AI R&D Acceleration at Anthropic in October 2026

Source: https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/

Published: 2026-11-05. Retrieved: 2026-11-08.

Primary source record. Findings retain this source's task, date, uncertainty and scope; the reference itself supplies no independent validation.

Learn After

Learn After