Comparison

More Confident Human Review Can Remain Wrong

Recchia et al.'s extended preprint (2025-05-17) tested people on difficult binary law, medical and Lojban-language questions. Study 1 found no overall accuracy benefit from initial opposing model-generated arguments before consultation, nor evidence that assisted humans outperformed the model alone; exploratory accuracy gains appeared when the model was wrong. Study 2's preregistered analyses found no significant overall differences with versus without model-generated research guidelines accompanying a single AI advisor. After roughly 20 minutes' web research, both groups increased confidence in that advisor's answer on average, including when wrong. Synthesis: confidence and correct checking are distinct outcomes. Simple protocols, often weak guidelines and Study 2's changed instructions/incentives limit generalization; these null results neither establish failure of all oversight nor test novel-discovery review.

Evidence status: bounded_primary_human_null_results_and_confidence_failure.

0

1

Updated 2026-11-08

Contributors are:

Who are from:

Tags

AI Scientific Oversight: Problem and Approaches

1Cademy