How can you tell a development set has been overused?
Question: In one to three sentences, explain what pattern shows that a development set has started to reflect the optimizer's choices rather than true model quality.
Sample answer: A warning sign is that the model scores much better on the set you keep checking than on a separate, untouched evaluation set. That mismatch means the repeated tuning is fitting the checked set's details instead of improving general performance.
Key points:
- Compare the score on the repeatedly consulted set with an independent evaluation set
- A sizable gap, with the consulted set looking better, indicates overfitting to that set
Rubric: Full credit if the answer identifies a strong score gap between the consulted development set and an independent holdout set as the sign of overfitting.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What does a much higher score on the Eyeball dev set than on the Blackbox dev set usually indicate?
True or False: If a small development set is being overfit, collecting additional data for that set can help address the problem.
If your tuning set looks much better than your holdout set, you have _____ the tuning set.
Match each dev set situation to its best interpretation.
Steps for spotting an overused validation set
Why adding more examples can reduce overfitting in a manually reviewed validation set
Diagnosing a reviewed dev set that has become too familiar
How can you tell a development set has been overused?
What is the recommended fix when a development set has become too specialized to be a reliable guide?
True or False: If a model scores higher on a small hand-reviewed validation set than on a separate hidden validation set, that always means the model is well tuned.