Question: A dev set contains 800 examples, and the model reaches 96.0% accuracy. Explain why teams often end up correcting labels mainly among the 32 examples the model misses instead of checking all 768 examples it classifies correctly.

Sample answer: At 96.0% accuracy, there are 32 mistakes and 768 correct predictions. It is much faster to inspect 32 cases than to review 768, so people naturally start with the smaller error set. That behavior is driven by time and effort, not by a deliberate decision that the error set is the only place worth checking. If corrections are made only in the smaller group, the dev set can become biased because many label problems in the larger correct group are never examined.

Key points:

• 800 examples at 96.0% accuracy means 32 misclassified and 768 correctly classified examples
• Reviewing 32 examples is far less work than reviewing 768
• The pattern comes from convenience, not an intentional labeling policy
• Correctly classified examples can still contain label errors that remain undiscovered

Rubric: Full credit requires stating the 32 versus 768 split, explaining why the smaller group is easier to inspect, and noting that the resulting bias is an unintended consequence of convenience.