Case context: A data team builds a model to predict whether a loan applicant will miss a payment. The development set contains mostly applicants with long, stable employment histories and bank-issued credit cards. In production, the evaluation set includes many freelancers and short-term contractors with irregular income patterns. The model scores 98% accuracy on the development set but only 73% on the evaluation set.

Question: Using the idea of mismatched development and evaluation sets, what is the most likely reason for the performance gap? What does this imply about the team's earlier tuning work on the development set?

Sample answer: The most likely cause is that the development and evaluation sets come from different distributions. The model was tuned on one type of applicant pattern, but the evaluation set reflects a different population. This means the lower evaluation performance does not necessarily show that the evaluation set is harder; it shows that the development-set improvements did not carry over well. As a result, much of the earlier tuning effort focused on the development set was probably poorly spent, because it improved performance on the development set without improving results on the evaluation set.

Key points:

• The development and evaluation sets are drawn from different distributions.
• The evaluation set is different, not automatically more difficult.
• Adjustments that helped on the development set did not transfer well.
• Earlier tuning work on the development set was likely wasted.

Rubric: The answer must identify the distribution mismatch and explain that prior tuning on the development set was likely wasted effort because the same improvements did not carry over to the evaluation set.