Case context: A machine learning team builds a system and evaluates it on their dev set, achieving strong performance. When they later evaluate the same system on their test set, performance is noticeably worse. The team confirms that both the dev and test sets were sampled from the same underlying data distribution.

Question: Based on this case, what should the team diagnose as the cause of the performance gap, and what should they do about it?

Sample answer: Since the dev and test sets come from the same distribution, the team should diagnose that their system has overfit the dev set; the performance gap is not due to the test set representing different data. The obvious cure is to get more dev set data, which will make the dev set a more reliable and representative basis for tuning the system, reducing the risk of the system fitting to idiosyncrasies of a small dev set.

Key points:

• Same distribution rules out distribution mismatch as a cause
• Diagnosis is that the system has overfit the dev set
• Cure is to obtain more dev set data
• The diagnosis is described as clear/obvious given the shared distribution condition

Rubric: Full credit requires identifying dev set overfitting as the diagnosis given the shared distribution, and recommending more dev set data as the cure.