Question: Explain why noisy or incorrect labels in a dev or test set may be acceptable early in a project but become a serious issue later. What changes in the model’s performance make label cleanup more valuable?

Sample answer: At the start of a project, the model usually makes many errors, so a handful of mislabeled dev examples is only a small part of the total mistakes. Those bad labels do not change the evaluation much when the overall error rate is still high. As the model improves, however, the total number of genuine errors shrinks. The mislabeled examples then make up a larger share of what remains, so they add more noise to the measured score. That noise makes it harder to tell whether a new model is truly better, especially when comparing close results such as 1.6% versus 1.9% error. Once the system is near that stage, cleaning the labels becomes worthwhile.

Key points:

• Early on, many real errors hide the effect of a few bad labels.
• As error rates fall, mislabeled examples become a larger fraction of the remaining mistakes.
• Bad labels create noise in evaluation numbers.
• Cleaning the dev set helps compare small performance gains reliably.

Rubric: The answer must explain that the decision changes because the proportion of label errors among all errors grows as model quality improves, making evaluation less trustworthy and label cleanup more useful.