Learn Before
What problem can occur if test-set scores are used to decide when to roll back a model version?
Question: If a team checks test-set performance every time it considers rolling back to an earlier model version, what is the main danger for the test set's value as an evaluation tool?
Sample answer: The main danger is that repeated use of the test set in decision-making will bias the team toward the test set. Over time, the test set stops being an independent check, so it can no longer provide a fully unbiased estimate of how the system will perform.
Key points:
- Repeated decisions based on the test set bias the team toward that set
- The test set loses its role as an independent evaluation
- Its performance estimate is no longer fully unbiased
Rubric: The answer should explain that using the test set to guide rollback decisions risks biasing the team toward the test set, which prevents it from remaining a fully unbiased estimate of performance.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main danger of using one holdout set over and over to decide releases, rollbacks, or model changes?
Which dataset should normally be used to compare model versions and decide whether to revert to an earlier checkpoint during development?
What happens if you repeatedly use the test set to choose among model variants?
When is it acceptable to check the test set during model development?
Using the test set to choose whether to keep a model rollback affects the test set's objectivity.
If you keep choosing models by looking at the test set, you begin to _____ the test set.
Classify Each Use of the Test Set as Safe or Unsafe
How Test-Set Feedback Can Make Evaluation Unreliable
Why should the test set stay out of model-tuning decisions?
Using the test set once a week to monitor overall progress is acceptable.
Once the test set has been used to compare models, it can no longer provide a fully _____ estimate of real-world performance.
Match each concept to its meaning when protecting a test set from being used during development.
Order the steps for using a test set without contaminating it during model development.
Why a weekly benchmark should not decide model rollbacks
Spotting the mistake in a model selection decision
What problem can occur if test-set scores are used to decide when to roll back a model version?