Overfitting a development set used for model tuning
If a model performs much better on the tuning set than on a separate final holdout set, the tuning set has probably been overused. The practical remedy is to collect more examples for that tuning set so it better represents the target distribution and is harder to memorize indirectly.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Overfitting a development set used for model tuning
What most clearly indicates that a manually inspected dev slice has been over-tuned during error analysis?
Inspecting a manually reviewed validation set can make you adapt to that set more quickly than if you never looked at its examples.
Keeping a Reviewed Subset Separate from a Hidden Check Set
Match each validation-set concept to its description in a reviewed-vs-reserved split.
Order the steps for checking whether repeated manual review is causing overfitting to a reviewed validation subset.
What is the most likely conclusion when manual review keeps pushing one dev set score far above another?
In the Eyeball/Blackbox approach, the Blackbox dev set is checked manually during routine error analysis.
If the Eyeball dev set improves much faster than the Blackbox dev set, you have _____ the Eyeball dev set.
Match each performance pattern with the most appropriate interpretation when comparing a hand-reviewed set with a separate hidden set.
Order the steps for checking whether a manually reviewed dev set has been overfit.
Detect Overfitting to an Inspected Development Set
When Manual Review Improves Faster Than a Hidden Evaluation Set
Why split the validation set into reviewed and unreviewed parts?
Learn After
What does a much higher score on the Eyeball dev set than on the Blackbox dev set usually indicate?
True or False: If a small development set is being overfit, collecting additional data for that set can help address the problem.
If your tuning set looks much better than your holdout set, you have _____ the tuning set.
Match each dev set situation to its best interpretation.
Steps for spotting an overused validation set
Why adding more examples can reduce overfitting in a manually reviewed validation set
Diagnosing a reviewed dev set that has become too familiar
How can you tell a development set has been overused?
What is the recommended fix when a development set has become too specialized to be a reliable guide?
True or False: If a model scores higher on a small hand-reviewed validation set than on a separate hidden validation set, that always means the model is well tuned.