Question: In a concise analytical response, explain why the training dev set should come from the training distribution but remain excluded from algorithm training.

Sample answer: Drawing the training dev set from the same distribution as the training set keeps the two subsets comparable. Withholding it from training provides separate data on which the team can evaluate and track learning progress. It therefore only needs to be large enough for those evaluation and tracking purposes and is usually smaller than the actual training set.

Key points:

• Drawn from the same distribution as the actual training set
• Not used to train the algorithm
• Used to evaluate and track learning progress
• Usually smaller than the training set
• Only needs to be large enough for evaluation and tracking

Rubric: A strong response identifies the shared distribution, explains that the subset is not used for training, connects this separation to evaluation and progress tracking, and accurately describes its size requirement.