Case context: A team has a dev set of 1,000 examples and a classifier with 98.0% accuracy. To save time, they only review and correct labels for the 20 examples the classifier misclassified, leaving the 980 correctly classified examples unreviewed.

Question: What bias does this practice introduce into the dev set, and why does it happen?

Sample answer: By only reviewing the 20 misclassified examples, the team fixes any label errors in that small subset while leaving potential label errors in the 980 correctly classified examples untouched. This happens purely because reviewing 20 examples is far more practical than reviewing 980. The result is a dev set where label corrections are applied unevenly across the misclassified and correctly classified groups, creating label-correction bias rather than a uniformly accurate label set.

Key points:

• Only the 20 misclassified examples receive label review and correction
• The 980 correctly classified examples remain unreviewed for label errors
• The cause is practical convenience, not a deliberate methodological decision
• The result is uneven, biased label correction across the dev set

Rubric: Full credit requires recognizing that only the misclassified subset gets corrected, that the correctly classified subset may still contain label errors, and that the underlying cause is practical convenience rather than a principled review strategy.