Case context: A team spent six weeks building a fraud-detection model for an online payments platform. During that time, they checked the same development set after nearly every change and adjusted features, thresholds, and network size to improve the development score. By the end of the project, the model scored 95% on the development set but only 79% on the test set.

Question: What evaluation problem should be identified in this scenario, and what should the team do next?

Sample answer: The model has likely overfit to the development set because the team repeatedly optimized against it over the six-week cycle. The large gap between development performance (95%) and test performance (79%) supports that diagnosis. The next step is to create a fresh development set and use it for future model selection.

Key points:

• Diagnose overfitting to the development set.
• Use the large development-test gap as evidence.
• Replace the dev set with a new one for future tuning.

Rubric: Full credit is given for identifying development-set overfitting caused by repeated evaluation and recommending a fresh development set.