Case context: A data science team builds a fraud detection model for an online marketplace. The model performs well on the development set, but its score drops noticeably on the final test set. The team confirms that the development and test sets were sampled from the same overall population and were created using the same labeling rules.

Question: Based on this case, what should the team conclude is causing the performance gap, and what should they do next?

Sample answer: Because the development and test sets come from the same distribution, the most likely explanation is that the model has fit too closely to the development set. The gap is not evidence of a distribution shift between dev and test. The practical response is to collect additional development examples so tuning decisions are based on a larger, more representative dev set and are less affected by chance quirks in a small sample.

Key points:

• Same distribution means distribution shift is not the explanation
• The model has likely overfit the development set
• The recommended fix is to gather more development data
• The conclusion follows directly from the shared source of the dev and test sets

Rubric: Full credit requires identifying development-set overfitting as the cause given the shared distribution, and recommending more development data as the remedy.