AI Debate with Human Judges: Task and Comparator Limits
Here, debate uses opposing AI advocates before a judge; consultancy uses one advisor. Rahman et al. (first submitted 2025-06-02; v2 2025-10-29) report final human accuracy of 70.1% after debate versus 60.0% after consultancy on curated COVID-19 claims, and 72.1% versus 68.3% on climate claims. COVID consultants' assigned positions were 50% correct; climate consultants chose positions, 92.5% correct. Judges lacked underlying sources. Carro et al.'s preprint (2026-10-20) reports within-subtopic human accuracy rising from 41.30% to 57.61% (adjusted p=.028), while consultancy showed no significant change; cross-subtopic evidence was inconclusive. Within-subtopic testing repeats the debated claim; cross-subtopic testing asks a different related claim, so pre/post items are not aligned. Its author-written claims lacked expert verification. Synthesis: task and comparator constrain interpretation. Selected nonexperts judged bounded factual claims; neither study tested communities validating or redirecting new AI-generated discoveries.
Evidence status: bounded_primary_human_findings_and_comparative_synthesis.
0
1
Tags
AI Scientific Oversight: Problem and Approaches
1Cademy