Case context: An engineering team builds a spam filter. They achieve 99% accuracy on their dev set, which consists of emails from corporate domains. However, when deployed to the test set, which contains a large proportion of personal emails from free providers, the accuracy drops to 70%.

Question: Based on the concept of mismatched dev/test sets, diagnose what is likely causing this discrepancy. What does this mean for the team's past optimization efforts on the corporate emails?

Sample answer: The discrepancy is caused by mismatched dev and test sets; the distributions are different (corporate vs. personal emails). The test set isn't inherently harder, but its different nature means the optimizations made for the dev set do not transfer. Consequently, a lot of the team's past effort to achieve 99% accuracy on the dev set was likely wasted, as those specific optimizations did not improve test set performance.

Key points:

• Dev and test sets have mismatched distributions.
• The test set is different, not necessarily harder.
• Dev set optimizations did not transfer to the test set.
• Past optimization effort on the dev set was likely wasted.

Rubric: The answer must identify the mismatch in distributions and state that previous optimizations on the dev set were likely wasted effort because what works on the dev set isn't working on the test set.