Question: A team uses a dev set to compare two image classifiers and choose the better one. Explain why simply discovering mislabeled dev examples is not enough to justify spending time correcting them.

Sample answer: A dev set is meant to support fast model comparison and selection. Because of that, the key issue is not whether any mistakes exist in the labels, but whether those mistakes are large enough to distort the comparison between models. If the mislabeled examples do not affect the decision about which classifier is better, fixing them may not be worth the effort. If they do interfere with that judgment, then correcting them becomes important because it restores the dev set's usefulness for choosing between models.

Key points:

• The dev set is used for quick model evaluation and selection.
• The presence of label errors alone does not determine whether they should be fixed.
• The deciding factor is whether the errors change or weaken the comparison between models.
• Label correction is worthwhile when the errors interfere with choosing the better algorithm.

Rubric: A strong response states the purpose of the dev set, explains that the effect of the mislabels matters more than their mere existence, and concludes that correction is justified when the errors affect model comparison.