Case context: A team has one pool of labeled examples collected from the same source. It splits that pool into two parts, trains its model on both parts, and then uses the smaller part to report a validation score.

Question: What is wrong with this plan, and how should the smaller part be used instead?

Sample answer: The smaller part is not a true holdout set because the model was trained on it. The team should keep one subset completely out of training, train only on the main training subset, and then use the untouched subset to measure progress and compare model versions. That holdout subset only needs to be large enough to serve that evaluation role.

Key points:

• A holdout used for evaluation must not be included in training
• The training and holdout parts should come from the same training distribution
• Model fitting should use only the main training subset
• The held-out subset is for evaluation and tracking progress
• The held-out subset only needs to be large enough for that purpose

Rubric: The response must identify training on the supposed holdout subset as the core error and recommend reserving a separate, same-distribution subset for evaluation and progress tracking.