Why can a validation-set gain be less trustworthy when the validation and test sets come from different distributions?
Question: Answer in one to three sentences. Explain how a distribution mismatch affects what a validation improvement tells you about test performance.
Sample answer: A gain on the validation set is measured on one distribution, while the test set comes from another. Because the two sets are not drawn from the same distribution, the validation improvement may not carry over to the test set, so the result is less reliable.
Key points:
- The validation and test distributions differ.
- Improvement on validation may not predict test improvement.
Rubric: The response should say that the distributions differ and that this creates uncertainty about whether validation-set gains will transfer to the test set.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What remains uncertain when the validation and test sets come from different distributions?
A change in development and test distributions can make it harder to choose which model issue to fix first.
Different dev and test distributions add extra uncertainty to model evaluation
Match each distribution issue to its downstream consequence.
Order the reasoning chain when the development set and test set come from different populations.
Why do different development and target data make model improvement harder to judge?
Explain why a stronger dev result may not justify the next engineering priority.
Why can a validation-set gain be less trustworthy when the validation and test sets come from different distributions?
Why does a dev/test distribution mismatch make it harder to choose fixes?
A higher score on a mismatched development set guarantees better test performance.