Case context: A team has a dev set of 5,000 examples with a 20% error rate, producing about 1,000 misclassified examples. Examining every error manually would take a long time.

Question: What dev-set arrangement should the team choose, and how should it use each portion?

Sample answer: The team should explicitly split the dev set into a portion it manually examines and a portion it does not examine. It should conduct error analysis on the examined portion and use the hands-off portion to tune parameters. Because the examined portion will be overfit more rapidly, comparing the portions can help the team detect overfitting caused by its manual error analysis.

Key points:

• Explicitly split the large dev set into two portions.
• Manually inspect only the examined portion.
• Keep the other portion free from manual inspection.
• Use the unexamined portion for parameter tuning.
• Use the separation to detect overfitting of the examined portion.

Rubric: The response should recommend an explicit two-part split, assign manual error analysis to the examined portion, assign parameter tuning to the unexamined portion, and state how the split helps detect overfitting.