Question: Suppose the data used to train a model comes from one distribution, while the data used for development and testing comes from another. Explain what each dataset in a four-set evaluation scheme is used to measure: the training set, the training-dev set, the dev set, and the test set.

Sample answer: The four-set scheme separates several kinds of evaluation. The training set is used to measure training error, so it shows how well the model fits the data it was optimized on. The training-dev set is drawn from the same distribution as the training set, but it is not used for fitting; it measures how well the system generalizes to fresh examples from that same training distribution. The dev set is used to judge performance on the real target distribution during model selection, and the test set provides the final check on that same target task.

Key points:

• Training set: measures training error.
• Training-dev set: measures generalization to new data from the training distribution.
• Dev set: measures performance for model selection on the target distribution.
• Test set: measures final performance on the target task.

Rubric: A strong response clearly assigns a distinct diagnostic role to each set, including training error for the training set, same-distribution generalization for the training-dev set, and target-task performance for the dev and test sets.