Google

When the test set is different from the dev set, work that improves dev set performance may not improve test set performance, so much of the dev-set optimization effort may be wasted.

Mismatched Dev/Test Sets Can Waste Dev-Set Optimization Effort

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Consequence of Mismatched Dev/Test Sets

Mismatched Dev/Test Sets Make Prioritization Harder

The Test Set is _____ from the Dev Set

Impacts of Mismatched Dev/Test Distributions

The Logical Chain of Wasted Optimization Effort

Question:
Explain why optimization effort on a dev set might be wasted if the dev and test sets are mismatched. In your answer, discuss how this situation affects a team's ability to prioritize their work.

Sample answer:
When dev and test sets are mismatched, the test set is different, though not necessarily harder. Consequently, algorithms or features that successfully improve performance on the dev set's specific distribution often do not translate to improvements on the test set. Because these improvements fail to generalize, the time and effort spent optimizing the dev set can be largely wasted. Furthermore, this mismatch introduces uncertainty; because it is unclear if dev set gains will yield real-world improvements, teams find it much harder to figure out what is genuinely working and how to prioritize their future engineering tasks.

Key points:
- The test set is different, not necessarily harder.
- What works on the dev set often does not translate to the test set.
- Dev-set optimization effort might be wasted.
- Introduces uncertainty about actual performance gains.
- Makes it significantly harder to figure out what is working and to prioritize.

Rubric:
A strong response will explicitly connect mismatched distributions to the failure of performance gains to generalize, leading to wasted effort. It must also identify that this uncertainty makes task prioritization more difficult.

Explaining Wasted Effort and Prioritization Challenges

Case context:
An engineering team builds a spam filter. They achieve 99% accuracy on their dev set, which consists of emails from corporate domains. However, when deployed to the test set, which contains a large proportion of personal emails from free providers, the accuracy drops to 70%.

Question:
Based on the concept of mismatched dev/test sets, diagnose what is likely causing this discrepancy. What does this mean for the team's past optimization efforts on the corporate emails?

Sample answer:
The discrepancy is caused by mismatched dev and test sets; the distributions are different (corporate vs. personal emails). The test set isn't inherently harder, but its different nature means the optimizations made for the dev set do not transfer. Consequently, a lot of the team's past effort to achieve 99% accuracy on the dev set was likely wasted, as those specific optimizations did not improve test set performance.

Key points:
- Dev and test sets have mismatched distributions.
- The test set is different, not necessarily harder.
- Dev set optimizations did not transfer to the test set.
- Past optimization effort on the dev set was likely wasted.

Rubric:
The answer must identify the mismatch in distributions and state that previous optimizations on the dev set were likely wasted effort because what works on the dev set isn't working on the test set.

Diagnosing the Spam Filter Performance Gap

Question:
Why does having mismatched dev and test sets make it harder for a machine learning team to prioritize what to work on next?

Sample answer:
It introduces uncertainty about whether improving the dev set distribution will also improve test set performance. This makes it harder to figure out what is actually working, complicating task prioritization.

Key points:
- Introduces uncertainty about transferring improvements.
- Makes it harder to figure out what is and isn't working.

Rubric:
The answer should address the uncertainty created by the mismatch and how it obscures what is and isn't working.

Learn Before

Related