Why Dev and Test Sets Should Match
Question: In one to three sentences, explain why a dev/test gap is easier to interpret when both sets come from the same distribution.
Sample answer: If the dev and test sets are drawn from the same distribution, a score gap between them is less likely to be caused by different data. That makes dev-set overfitting the most plausible explanation for the difference.
Key points:
- Matching distributions removes data shift as an alternative explanation.
- The remaining gap can then be attributed to overfitting on the dev set.
Rubric: The answer should explain that using the same distribution for dev and test removes distribution mismatch as a competing cause and makes dev-set overfitting the clear diagnosis.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
A model does well on the validation set but performs poorly on the final test set, and both sets come from the same process. What is the most likely diagnosis?
True or False: If validation and test data come from the same source, a large gap between validation and test scores always proves the model has too many layers.
The practical fix for overfitting the validation set is to gather more _____ examples.
Match each performance pattern to its best diagnosis
Order the steps for diagnosing a validation/holdout performance gap from the same data source
Why a Shared-Distribution Validation/Test Gap Signals Validation Overfitting
Diagnose a dev/test gap when both evaluation sets come from the same source
Why Dev and Test Sets Should Match
What is the practical fix when a model has fit the development set too closely?
A gap between validation and test performance always means you should add more test examples.