Why does a dev/test distribution mismatch weaken confidence in a dev-set improvement?
Question: Answer in one to three sentences using the relationship between the dev and test distributions.
Sample answer: A dev-set improvement is measured on the dev distribution, which differs from the test distribution. Therefore, the team has additional uncertainty about whether the improvement will also raise test-set performance.
Key points:
- The dev and test distributions are mismatched.
- Improvement on the dev distribution may not transfer to test performance.
Rubric: The answer should state that the distributions differ and that this creates uncertainty about transferring dev-set gains to the test set.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What uncertainty arises when the dev and test sets have different distributions?
A mismatched dev and test distribution can make project priorities harder to determine.
Mismatched dev and test sets introduce additional _____ about whether dev gains transfer to test performance.
Match each distribution-related condition with its stated consequence.
Order the reasoning chain caused by mismatched dev and test distributions.
Explain why mismatched dev and test distributions complicate both evaluation and prioritization.
Diagnose why a team cannot confidently prioritize after its dev score improves.
Why does a dev/test distribution mismatch weaken confidence in a dev-set improvement?
Which reasoning best explains why mismatched dev and test sets hinder prioritization?
Improving on a mismatched dev distribution proves that test-set performance will also improve.