When Repeated Validation Checks Distort Model Selection
If a team keeps using the dev set to judge many ideas during development, the model can gradually become tuned to that specific split. A strong dev-set score at the end, paired with a noticeably worse test-set score, is a warning sign. In that situation, the safest response is to create a new dev set.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
When Evaluation Data Does Not Match Deployment Data
When Repeated Validation Checks Distort Model Selection
When the Metric Rewards the Wrong Goal
When should your validation setup be revised?
True or False: If your initial validation split or evaluation metric turns out to be poorly chosen, you cannot revise it without abandoning the project.
If your evaluation metric no longer reflects your main objective, what should you change?
What is the clearest sign that your dev/test set or evaluation metric may need revision?
If a validation set or metric turns out to be poorly matched to the real goal, the team should rebuild the whole project before making any changes.
What to revise when the evaluation no longer matches the goal
Match each reason a validation metric can mislead the team to the recommended remedy.
What should a team do when its evaluation setup stops matching its goal?
When Validation Data Does Not Match Deployment Data
After revising your dev/test sets or evaluation metric, updating the project documentation is enough; the team does not need to be told about the new direction.
What should be expanded after repeated tuning to the validation set?
Match each situation to the underlying problem category it illustrates.
Order the reasoning steps for deciding whether to replace an evaluation metric that no longer matches the product goal.
When validation results stop matching the best product choice
When Evaluation Scores and Product Needs Disagree
What should a team do after the development set stops guiding decisions?
Learn After
Keep the Test Set Out of Routine Model Decisions
What makes a model gradually adapt itself to the dev set during development?
A large gap with dev performance much better than test performance can indicate overfitting to the dev set.
If repeated evaluation has made the old development set misleading, get a _____ development set.
Match each dev-set overfitting concept to its description.
Put the dev-set overfitting process in the correct order.
After several rounds of tuning, your validation score is much better than your test score. What should you do next?
It is a good practice to keep checking the test set after every major training change so you can choose the best model.
Repeatedly choosing models based on dev-set results can cause the system to gradually _____ to the dev set.
Match each development-set practice or signal with what it means in model selection.
Order the steps for investigating a validation-versus-test performance gap.
How Repeated Tuning Can Distort the Development Set
Evaluating a Model That Fits the Development Set Too Closely
Sign That the Development Set Has Been Overused