Case context: A project team has a validation set of 8,000 examples. Its model gets 15% of them wrong, so there are about 1,200 errors to study. Reviewing every mistaken case by hand would be slow.

Question: What split of the validation set is most useful, and what should the team do with each part?

Sample answer: The team should divide the validation set into two clearly separated parts: one part for close human review and one part that is never used for that review. The reviewed part should be used to inspect failure patterns and identify common error sources. The untouched part should be reserved for choosing model settings and tracking performance. If the team sees that improvements on the reviewed part stop translating well to the untouched part, that is a sign it may be fitting too closely to the reviewed slice.

Key points:

• Make one explicit split between a reviewed subset and a reserved subset.
• Use the reviewed subset for hands-on error inspection.
• Keep the reserved subset free of that manual review.
• Use the reserved subset for model selection and parameter tuning.
• Compare the two subsets to spot overfitting to the reviewed examples.

Rubric: The response should recommend a clear two-part split, assign manual error inspection to one part, assign model selection or tuning to the other part, and explain that the separation helps reveal overfitting to the reviewed subset.