Case context: You are building a model that classifies customer support tickets into urgency levels. Early in development you noticed a few mislabeled dev examples, but you postponed fixing them. After several rounds of improvement, the model's overall dev error is now very low. When you inspect 100 of the remaining dev errors, you find that about 28 are due to incorrect labels in the dev set rather than model mistakes.

Question: What should the team do next with the dev set, and why does the current error breakdown support that choice?

Sample answer: The team should pause and clean the dev set labels. At this stage the model is already making relatively few mistakes, so mislabeled examples now account for a large share of the observed errors. That label noise makes the dev metric less trustworthy and can hide real differences between models that may differ by only a small amount.

Key points:

• Recommend relabeling the dev set.
• Note that mislabeled examples now represent a substantial fraction of the remaining errors.
• Explain that noisy labels make performance estimates unreliable.

Rubric: The response should recommend fixing the mislabeled dev set and justify the decision by stating that label noise now makes up a large portion of the remaining errors and therefore distorts evaluation.