Google

If dev and test sets come from different distributions, a system that works well on the dev set but poorly on the test set has several possible explanations: dev-set overfitting, a harder test set, or a test set that is different rather than harder.

Different Dev/Test Distributions Make Failure Diagnosis Ambiguous

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When the test set is different from the dev set, work that improves dev set performance may not improve test set performance, so much of the dev-set optimization effort may be wasted.

Mismatched Dev/Test Sets Can Waste Dev-Set Optimization Effort

Which is NOT listed in Machine Learning Yearning as a possible cause when a model does well on dev but poorly on the test set (different distributions)?

True or False: When dev and test sets come from different distributions, diagnosing why a model underperforms on the test set is straightforward.

Machine Learning Yearning warns that if dev and test sets come from different _____, a gap in performance leaves the cause of failure unclear.

Match each possible failure cause (when dev and test distributions differ) to its correct description from Machine Learning Yearning.

Order the three possible failure causes as they appear in Machine Learning Yearning when a model succeeds on dev but fails on test with mismatched distributions.

According to Machine Learning Yearning, what is the key implication if the test set is harder than the dev set when the two sets have different distributions?

True or False: According to Machine Learning Yearning, a lower test-set score compared to dev always means the test set is objectively harder.

Machine Learning Yearning states: 'So what works well on the _____ set just does not work well on the test set.'

Match each failure diagnosis (mismatched dev/test distributions) to the corrective implication it would suggest for a practitioner.

Order the reasoning steps that lead a practitioner to recognize diagnostic ambiguity when dev and test sets come from different distributions.

Question:
In machine learning projects where the dev and test sets are drawn from different distributions, a model might perform exceptionally well on the dev set but exhibit poor performance on the test set. Discuss the three possible explanations for this performance gap as identified in Machine Learning Yearning, and explain why this scenario leaves your options for optimization unclear.

Sample answer:
When a model works well on the dev set but poorly on a test set from a different distribution, there are three primary explanations. First, the model may have overfit to the dev set. Second, the test set might simply be harder than the dev set, which could mean the algorithm is already doing as well as can be expected. Third, the test set may not be harder, but merely different, meaning the features that work for the dev distribution do not apply to the test distribution. This creates diagnostic ambiguity because it is unclear whether the team should focus on reducing overfitting, improving general capabilities, or collecting more relevant data.

Key points:
- The model overfit to the dev set.
- The test set is harder than the dev set.
- The test set is just different from the dev set.
- It is unclear which problem to fix to improve performance.

Rubric:
The essay should correctly identify the three potential causes (overfitting to dev, harder test set, different test set) and explain how the inability to distinguish between them makes the path to further improvement unclear.

Diagnostic Ambiguity with Mismatched Dev/Test Distributions

Case context:
Your team has built a system that achieves excellent accuracy on your dev set. However, when evaluating it on the test set, the performance drops significantly. You are aware that your dev and test sets were drawn from entirely different distributions.

Question:
Based on Machine Learning Yearning, what are the three possible things that could have gone wrong in this scenario? Furthermore, in which of these three scenarios might you conclude that no further significant improvement is possible?

Sample answer:
The three possible things that could have gone wrong are: 1) the model overfit to the dev set, 2) the test set is harder than the dev set, or 3) the test set is just different from the dev set. If the test set is simply harder than the dev set, you might conclude that your algorithm is already doing as well as could be expected, and no further significant improvement is possible.

Key points:
- Overfit to the dev set
- Test set is harder
- Test set is different
- No further improvement might be possible if the test set is harder.

Rubric:
The response must list the three specific causes for the performance drop and correctly attribute the potential impossibility of further improvement to the scenario where the test set is inherently harder.

Troubleshooting a Performance Drop on the Test Set

Question:
According to Machine Learning Yearning, what are the three possible explanations when a system performs well on a dev set but poorly on a test set drawn from a different distribution?

Sample answer:
The three possible explanations are that the system overfit to the dev set, the test set is harder than the dev set, or the test set is simply different from the dev set.

Key points:
- Overfit to dev set
- Test set is harder
- Test set is different

Rubric:
The answer must list all three explanations: overfitting to the dev set, a harder test set, and a different test set.

Learn Before

Related