Case context: A machine learning team has a development set of 6,000 examples. They want to study errors by looking directly at some misclassified cases, but they also want their final performance measure to stay objective and not be influenced by which examples they inspect by hand.

Question: Given this situation, what split should the team make in the development set? Name the two parts, explain what each part is for, and describe which part may be examined manually and which part should remain untouched.

Sample answer: The team should divide the development set into two subsets: an Inspection set and a Locked set. The Inspection set is used for manual error analysis, so the team can look at specific mistakes and identify patterns. The Locked set is reserved for objective evaluation and should not be examined example by example, so it remains unbiased.

Key points:

• Split the development set into an Inspection set and a Locked set.
• Use the Inspection set for manual review and error analysis.
• Keep the Locked set untouched so it can support objective evaluation.

Rubric: The response must propose splitting the development set into an Inspection set and a Locked set, specify that the Inspection set is manually examined for error analysis, and specify that the Locked set must not be manually examined to preserve objective evaluation.