Example

TinyGenius: Agreement Depended on the Kind of Judgment

In TinyGenius, a microtask is one small judgment: does a machine-made label or summary fit supplied context, usually a passage from a paper’s brief overview? Oelen and colleagues (April 5, 2023) recruited 11 readily available computer-science researchers; each judged 114 statements randomly chosen within 10 selected popular machine-learning papers. Their 1,254 votes—not people—included 122 unknowns, excluded from agreement calculations; choices were correct, incorrect or unknown. Topic checks drew more agreement than sorting terms into categories. Summaries were rated relatively straightforward yet drew poor agreement. The authors suggested right/wrong judgments poorly fit partly adequate summaries and proposed an untested slider for future graded ratings. This small study measured perceived ease and agreement, not independent accuracy, learning gains or the papers’ scientific correctness.

0

1

Updated 2026-11-09

Contributors are:

Who are from:

Tags

Scientific Oversight of AI-Assisted Research

1Cademy