Four Evaluation Results Answer Different Questions
Case-based inference (synthesis): these results cannot be added as interchangeable evidence of oversight. Lean acceptance is relative to a stated proposition and premises, whereas METR's uncertain, partly redacted growth estimate describes change in task capability at one lab. Even the two model-focused tests differ: MATH reward models rank candidate solutions; the evaluation measures selected final-answer correctness; circuit tracing investigates internal pathways, incompletely. Consequently, a better MATH selection score cannot fill METR's publicly missing evidence or establish that a written explanation matches an internal pathway; a traced pathway cannot substitute for Lean's proof obligation. Combining these results into one oversight metric requires a justified mapping between those different targets, which these cases do not provide. None directly measures a scientist's understanding of unfamiliar research.
Evidence status: analytical_comparison_of_distinct_measurement_targets.
0
1
Tags
AI Scientific Oversight: Problem and Approaches
1Cademy