Comparison

Four Evaluation Results Answer Different Questions

Case-based inference (synthesis): these results cannot be added as interchangeable evidence of oversight. Lean acceptance is relative to a stated proposition and premises, whereas METR's uncertain, partly redacted growth estimate describes change in task capability at one lab. Even the two model-focused tests differ: MATH reward models rank candidate solutions; the evaluation measures selected final-answer correctness; circuit tracing investigates internal pathways, incompletely. Consequently, a better MATH selection score cannot fill METR's publicly missing evidence or establish that a written explanation matches an internal pathway; a traced pathway cannot substitute for Lean's proof obligation. Combining these results into one oversight metric requires a justified mapping between those different targets, which these cases do not provide. None directly measures a scientist's understanding of unfamiliar research.

Evidence status: analytical_comparison_of_distinct_measurement_targets.

0

1

Updated 2026-11-08

Contributors are:

Who are from:

Tags

AI Scientific Oversight: Problem and Approaches

1Cademy