Keep the Test Set Out of Routine Model Decisions
It is reasonable to run the test set at regular intervals, such as every week or every month, to report overall progress. It should not be used to choose model changes, tune parameters, or decide whether to revert to an earlier version. If those decisions are made from test-set results, the model begins to fit the test set itself, and the test set no longer provides an unbiased estimate of real-world performance.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Keep the Test Set Out of Routine Model Decisions
What makes a model gradually adapt itself to the dev set during development?
A large gap with dev performance much better than test performance can indicate overfitting to the dev set.
If repeated evaluation has made the old development set misleading, get a _____ development set.
Match each dev-set overfitting concept to its description.
Put the dev-set overfitting process in the correct order.
After several rounds of tuning, your validation score is much better than your test score. What should you do next?
It is a good practice to keep checking the test set after every major training change so you can choose the best model.
Repeatedly choosing models based on dev-set results can cause the system to gradually _____ to the dev set.
Match each development-set practice or signal with what it means in model selection.
Order the steps for investigating a validation-versus-test performance gap.
How Repeated Tuning Can Distort the Development Set
Evaluating a Model That Fits the Development Set Too Closely
Sign That the Development Set Has Been Overused
Learn After
What is the main danger of using one holdout set over and over to decide releases, rollbacks, or model changes?
Which dataset should normally be used to compare model versions and decide whether to revert to an earlier checkpoint during development?
What happens if you repeatedly use the test set to choose among model variants?
When is it acceptable to check the test set during model development?
Using the test set to choose whether to keep a model rollback affects the test set's objectivity.
If you keep choosing models by looking at the test set, you begin to _____ the test set.
Classify Each Use of the Test Set as Safe or Unsafe
How Test-Set Feedback Can Make Evaluation Unreliable
Why should the test set stay out of model-tuning decisions?
Using the test set once a week to monitor overall progress is acceptable.
Once the test set has been used to compare models, it can no longer provide a fully _____ estimate of real-world performance.
Match each concept to its meaning when protecting a test set from being used during development.
Order the steps for using a test set without contaminating it during model development.
Why a weekly benchmark should not decide model rollbacks
Spotting the mistake in a model selection decision
What problem can occur if test-set scores are used to decide when to roll back a model version?