Why must dev and test sets share the same distribution to clearly diagnose dev set overfitting?
Question: In one to three sentences, explain why dev and test sets must come from the same distribution for a dev/test performance gap to clearly indicate dev set overfitting.
Sample answer: If dev and test sets come from the same distribution, any performance gap cannot be blamed on the test data being different, leaving dev set overfitting as the clear explanation. If the distributions differed, the gap could instead be due to the test set representing different data, making the diagnosis unclear.
Key points:
- Same distribution removes distribution mismatch as a competing explanation
- This isolates dev set overfitting as the clear diagnosis
Rubric: Answer should note that shared distribution eliminates distribution mismatch as an alternative explanation, isolating overfitting as the cause.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
A system performs well on the dev set but poorly on the test set (same distribution). What is the diagnosis?
True or False: If dev and test sets share a distribution, dev-test performance gaps always mean the model is too complex.
The obvious cure for overfitting the dev set is to get more _____ data.
Match each dev/test performance scenario to its correct diagnosis
Order the reasoning steps for diagnosing a same-distribution dev/test performance gap
Explain why a same-distribution dev/test gap points specifically to dev set overfitting
Diagnose a team's dev/test performance gap in a shared-distribution setting
Why must dev and test sets share the same distribution to clearly diagnose dev set overfitting?
What is the recommended cure when a system has overfit the dev set?
True or False: A dev/test performance gap always requires collecting more test set data as the fix.