Learn Before
Spotting the mistake in a model selection decision
Case context: A product team evaluates its machine translation system on the test set at the end of every month to track progress. In the latest review, the test score is slightly lower than last month. The team lead immediately tells everyone to discard the new system and return to the previous release.
Question: What methodological mistake did the team make, and what is the long-term effect on the trustworthiness of the test results?
Sample answer: The mistake was using the test set to make a system-selection decision. Once the test set influences choices such as rollback or model selection, the team starts to overfit to that test set. Over time, the test set stops being a fully unbiased estimate of real performance.
Key points:
- The team used test performance to decide whether to keep or reject a system version.
- That turns the test set into a decision tool, which causes overfitting to it.
- After repeated use this way, the test set is no longer a reliable unbiased estimate of actual performance.
Rubric: The learner must identify that the test set was used for a model or system decision and explain that this creates overfitting and undermines the test set's role as an unbiased performance estimate.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main danger of using one holdout set over and over to decide releases, rollbacks, or model changes?
Which dataset should normally be used to compare model versions and decide whether to revert to an earlier checkpoint during development?
What happens if you repeatedly use the test set to choose among model variants?
When is it acceptable to check the test set during model development?
Using the test set to choose whether to keep a model rollback affects the test set's objectivity.
If you keep choosing models by looking at the test set, you begin to _____ the test set.
Classify Each Use of the Test Set as Safe or Unsafe
How Test-Set Feedback Can Make Evaluation Unreliable
Why should the test set stay out of model-tuning decisions?
Using the test set once a week to monitor overall progress is acceptable.
Once the test set has been used to compare models, it can no longer provide a fully _____ estimate of real-world performance.
Match each concept to its meaning when protecting a test set from being used during development.
Order the steps for using a test set without contaminating it during model development.
Why a weekly benchmark should not decide model rollbacks
Spotting the mistake in a model selection decision
What problem can occur if test-set scores are used to decide when to roll back a model version?