Question: In the context of improving label quality in a dev set, analyze the potential drawbacks of limiting your review exclusively to examples that your learning algorithm misclassified. Explain why reviewing correctly classified examples is also recommended.

Sample answer: Limiting the review to only misclassified examples can miss hidden errors. It is possible that both the original label assigned to an example and the learning algorithm's prediction were simultaneously wrong. If you only review misclassified examples, you will miss instances where the algorithm made an incorrect prediction that coincidentally matched an incorrect original label. Double-checking correctly classified examples helps ensure that the ground truth itself is accurate, rather than just agreeing with a potentially flawed algorithm.

Key points:

• Only checking misclassified examples misses hidden label errors.
• Both the original label and the algorithm's prediction can be wrong on the same example.
• Reviewing correctly classified examples verifies the accuracy of the ground truth labels.

Rubric: The response should clearly identify that focusing only on misclassified examples misses a specific category of error. It must explicitly state the scenario where both the original label and the algorithm's prediction are incorrect but match each other. The answer should emphasize that checking correctly classified examples ensures the ground truth is actually correct.