Diagnosing a drop in test set performance with mismatched distributions.
Case context: A team builds a speech recognition system. They optimize their model's parameters and achieve excellent performance on their development set. However, when they evaluate the final model on their test set, they observe a significant drop in accuracy. They then check their data sources and find that the dev set and test set were collected from different distributions.
Question: Given that the dev and test sets come from different distributions, explain why the team cannot make a definitive diagnosis about why the model failed on the test set, and list the possible explanations they must consider.
Sample answer: Because the dev and test sets come from different distributions, the diagnosis of the poor test set performance is ambiguous. The team cannot definitively isolate the issue. They must consider three possibilities: they may have overfit the dev set, the test set may simply be harder than the dev set, or their algorithm might be performing as well as could be expected.
Key points:
- Different dev and test distributions make diagnosing poor test performance ambiguous.
- The team may have overfit the model to the dev set.
- The test set might be harder than the dev set.
- The algorithm might be doing as well as could be expected.
Rubric: The answer should explain that different distributions make the diagnosis ambiguous or unclear. It must list the three possible explanations grounded in the source text: 1) Overfitting the dev set, 2) The test set being harder than the dev set, and 3) The algorithm doing as well as could be expected.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Dev Set Should Reflect the Task to Improve Most
Same-Distribution Dev/Test Failure Indicates Dev Set Overfitting
Different Dev/Test Distributions Make Failure Diagnosis Ambiguous
Third-Party Benchmark Distribution Mismatch Increases Luck
When dev and test sets share the same distribution and test performance is worse than dev performance, what does this clearly indicate?
True or False: When dev and test sets come from different distributions, a performance gap between them has a single, unambiguous diagnosis.
When a system has overfit the dev set and both sets share the same distribution, the obvious cure is to get more _____ data.
Why should the dev set reflect the task a team wants to improve on the most?
If both sets share the same distribution and a model performs well on dev but poorly on test, the clear diagnosis is dev set overfitting.
When a model overfits the dev set and both sets share the same distribution, the obvious cure is to get more _____ data.
Match each dev/test set scenario to its consequence for model diagnosis.
Order the diagnostic steps when a model works well on the dev set but fails on the test set.
Which is a possible explanation for poor test performance when dev and test sets come from different distributions?
When dev and test sets come from different distributions, a system's failure on the test set provides an unambiguous diagnosis.
Once the dev and test sets are defined, a team will be focused on improving _____ set performance.
Match each concept related to dev/test distribution to its correct description.
Order the steps for selecting dev and test sets that support clear model evaluation.
Compare the diagnostics of poor test performance under same vs. different dev/test distributions.
Diagnosing a drop in test set performance with mismatched distributions.
Identify the diagnosis and cure for poor test performance when distributions match.