Case context: A team is building a classifier from one data source, but the data they ultimately care about comes from a different source. They keep four splits: a fit set, a holdout from the same source as the fit set, a development set, and a final test set.

Question: In this setup, what does checking the model on the fit set, the same-source holdout, and the development/test data tell you?

Sample answer: Performance on the fit set tells you the model's training error. Performance on the holdout from the same source tells you how well the model handles new examples that still match the training distribution. Performance on the development or test data tells you how well the model is likely to do on the real problem it will face after deployment.

Key points:

• Fit set evaluation measures training error.
• Same-source holdout evaluation measures generalization to new data from the training distribution.
• Development/test evaluation measures performance on the deployment task.

Rubric: The response must correctly identify that: 1. Fit set evaluation measures training error. 2. Same-source holdout evaluation measures generalization to new data from the training distribution. 3. Development/test evaluation measures performance on the deployment task.