Three Reasons a Held-Out Test Set Can Look Worse
Question: A model performs well on a tuning/validation set but much worse on a final test set that comes from another source. What three broad explanations should you consider?
Sample answer: The three main explanations are that the model was tuned too closely to the validation set, the final test set is genuinely more difficult, or the final test set comes from a different distribution than the validation set.
Key points:
- Overfitting to the validation set
- Final test set is harder
- Final test set is different
Rubric: The answer must name all three explanations: overfitting to the validation set, a harder test set, and a different test set.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why a Mismatched Validation Set Can Waste Effort
Which explanation is NOT one of the standard reasons a model can score well on the dev set but poorly on the test set when the two sets come from different distributions?
True or False: If a development set and a test set come from different distributions, it is easy to pinpoint the reason a model scores worse on the test set.
If the development set and the test set come from different _____, a performance gap is hard to interpret.
Match each distribution-shift situation to the most likely explanation.
Order the three explanations for why a model can look strong on development data but weak on test data when the data distributions are different.
When the test data are drawn from a tougher distribution than the development data, what is the most reasonable conclusion?
True or False: If a model scores lower on the test set than on the dev set, then the test set must be inherently more difficult.
A model may do well on the _____ set and still perform poorly on the test set if the two sets come from different distributions.
Match each dev/test diagnosis to the most appropriate next step.
Arrange the steps a practitioner follows when a spam filter succeeds on validation data but fails after deployment.
Why a good development score can still leave test results unexplained
Why a Model Looks Strong in Validation but Weak on Deployment Data
Three Reasons a Held-Out Test Set Can Look Worse