Comparison

AI Debate with Human Judges: Task and Comparator Limits

Here, debate uses opposing AI advocates before a judge; consultancy uses one advisor. Rahman et al. (first submitted 2025-06-02; v2 2025-10-29) report final human accuracy of 70.1% after debate versus 60.0% after consultancy on curated COVID-19 claims, and 72.1% versus 68.3% on climate claims. COVID consultants' assigned positions were 50% correct; climate consultants chose positions, 92.5% correct. Judges lacked underlying sources. Carro et al.'s preprint (2026-10-20) reports within-subtopic human accuracy rising from 41.30% to 57.61% (adjusted p=.028), while consultancy showed no significant change; cross-subtopic evidence was inconclusive. Within-subtopic testing repeats the debated claim; cross-subtopic testing asks a different related claim, so pre/post items are not aligned. Its author-written claims lacked expert verification. Synthesis: task and comparator constrain interpretation. Selected nonexperts judged bounded factual claims; neither study tested communities validating or redirecting new AI-generated discoveries.

Evidence status: bounded_primary_human_findings_and_comparative_synthesis.

0

1

Updated 2026-11-08

Contributors are:

Who are from:

Tags

AI Scientific Oversight: Problem and Approaches

1Cademy