Sign That the Development Set Has Been Overused
Question: After you finish training and selecting your final model, what comparison suggests that you have effectively overfitted the development set?
Sample answer: A warning sign is when the model performs much better on the development set than on the test set, which suggests the development data influenced the model-selection process too strongly.
Key points:
- Compare development-set performance with test-set performance.
- A large gap, with development performance much better, indicates overfitting to the development set.
Rubric: The answer must state that development-set performance should be compared with test-set performance and that a much better development-set result is the sign of overfitting.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Keep the Test Set Out of Routine Model Decisions
What makes a model gradually adapt itself to the dev set during development?
A large gap with dev performance much better than test performance can indicate overfitting to the dev set.
If repeated evaluation has made the old development set misleading, get a _____ development set.
Match each dev-set overfitting concept to its description.
Put the dev-set overfitting process in the correct order.
After several rounds of tuning, your validation score is much better than your test score. What should you do next?
It is a good practice to keep checking the test set after every major training change so you can choose the best model.
Repeatedly choosing models based on dev-set results can cause the system to gradually _____ to the dev set.
Match each development-set practice or signal with what it means in model selection.
Order the steps for investigating a validation-versus-test performance gap.
How Repeated Tuning Can Distort the Development Set
Evaluating a Model That Fits the Development Set Too Closely
Sign That the Development Set Has Been Overused