Question: Explain why optimization effort on a dev set might be wasted if the dev and test sets are mismatched. In your answer, discuss how this situation affects a team's ability to prioritize their work.

Sample answer: When dev and test sets are mismatched, the test set is different, though not necessarily harder. Consequently, algorithms or features that successfully improve performance on the dev set's specific distribution often do not translate to improvements on the test set. Because these improvements fail to generalize, the time and effort spent optimizing the dev set can be largely wasted. Furthermore, this mismatch introduces uncertainty; because it is unclear if dev set gains will yield real-world improvements, teams find it much harder to figure out what is genuinely working and how to prioritize their future engineering tasks.

Key points:

• The test set is different, not necessarily harder.
• What works on the dev set often does not translate to the test set.
• Dev-set optimization effort might be wasted.
• Introduces uncertainty about actual performance gains.
• Makes it significantly harder to figure out what is working and to prioritize.

Rubric: A strong response will explicitly connect mismatched distributions to the failure of performance gains to generalize, leading to wasted effort. It must also identify that this uncertainty makes task prioritization more difficult.