Case context: A product team maintains a validation set of 2,400 labeled examples. Their classifier reaches 97.5% accuracy, so about 60 examples are predicted incorrectly. To speed up review, the team checks and fixes labels only for those 60 error cases and does not inspect the remaining 2,340 predictions.

Question: What bias does this create in the validation set, and what is the reason?

Sample answer: This approach creates label-correction bias. The examples that the model gets wrong are the only ones that receive a label audit, so any labeling mistakes among them are more likely to be found and repaired. The much larger group of correctly predicted examples is left alone, even though some of those labels could still be wrong. Because the review effort is driven by convenience and limited time, the corrections are applied unevenly instead of being distributed across the full set.

Key points:

• Only the 60 error cases are checked for label problems
• The 2,340 predicted-correct cases may still contain bad labels
• The review strategy is chosen for efficiency, not to sample the whole set fairly
• The result is uneven correction across the validation data, which introduces bias

Rubric: Full credit requires stating that only the mispredicted subset is reviewed and corrected, that the unreviewed correctly predicted subset may still have label mistakes, and that the unequal review pattern arises from practical convenience rather than a balanced auditing plan.