Case context: An engineering team is building a model to classify bird species from photos. During a review, they discover that about 8% of the development examples contain wrong species labels. They pay specialists to review and fix every label in the development set. To reduce costs, they leave the test set untouched. After training, the model reaches 97% accuracy on the revised development set, but only 89% on the test set.

Question: What explains the gap between the two scores, and what should the team do so the evaluation is trustworthy?

Sample answer: The two scores are not directly comparable because the development set and the test set are no longer using the same labeling standard. The model was tuned and selected against the cleaned development labels, but the test score is still based on the original, uncorrected labels. To make the evaluation fair, the team should use the same label-review and correction procedure on the test set so both datasets are judged with the same target labels.

Key points:

• Recognize that only one dataset was relabeled, so the evaluation setup is inconsistent.
• Note that model selection happened against cleaned labels while the test score used noisy labels.
• Recommend applying the same correction process to the test set before comparing results.

Rubric: The response must state that the development and test sets are being judged under different label conditions because only the development labels were fixed. It must recommend applying the same label-correction process to the test set so the evaluation criteria match.