Case context: A team divides its available training data into two subsets drawn from the same distribution. It trains the algorithm on both subsets, then reports performance on the smaller subset as training-dev performance.

Question: What should the team diagnose as incorrect, and how should it revise its use of the smaller subset?

Sample answer: The smaller subset is not functioning as a training dev set because the algorithm trained on it. The team should keep a separate subset from the same training distribution out of training, train only on the actual training set, and use the held-out subset to evaluate and track learning progress. That subset only needs to be large enough for those purposes.

Key points:

• Training dev data must not be used for training
• The two subsets should share the training distribution
• Training occurs only on the actual training set
• The held-out subset supports evaluation and progress tracking
• The held-out subset need only be large enough for its evaluation role

Rubric: The response must diagnose training on the proposed training dev set as the central error and prescribe a separate, same-distribution subset withheld for evaluation and progress tracking.