Question: Explain why, when dev and test sets come from the same distribution, a system doing well on dev but poorly on test leads to the clear diagnosis of dev set overfitting, and why more dev data is the obvious cure.

Sample answer: When dev and test sets are drawn from the same distribution, any difference in performance between them cannot be attributed to the data being different in nature; both sets represent the same underlying data distribution. Therefore, if a system performs well on dev but poorly on test, the only remaining explanation is that the system has tuned itself too closely to the specific examples in the dev set rather than learning patterns that generalize to the shared distribution, which is overfitting the dev set. Since the problem is that the dev set is too small or too specific for reliable tuning, the obvious cure is to get more dev set data, giving the system a larger and more representative sample to tune against and reducing the chance of overfitting to a small dev set.

Key points:

• Same distribution eliminates distribution mismatch as an explanation for the gap
• Good dev performance with poor test performance under this condition indicates dev set overfitting
• Overfitting the dev set is a clear, specific diagnosis, not a vague or unclear one
• The obvious cure is to get more dev set data

Rubric: Full credit requires explaining that same distribution rules out distribution mismatch as an explanation, correctly identifying dev set overfitting as the diagnosis, and connecting more dev data as the appropriate cure.