Learn Before
Why a weekly benchmark should not decide model rollbacks
Question: A team checks a held-out benchmark every Friday to monitor progress. Explain why using that same score to decide whether to keep the current model or revert to an earlier version is a bad practice, and describe what happens to the quality of the benchmark estimate.
Sample answer: If the benchmark is used to choose between models, it becomes part of the decision-making loop instead of serving only as a final check. Repeatedly using it in this way encourages the team to adapt the model to that benchmark, which makes the system overfit to the benchmark itself. Once that happens, the benchmark can no longer be trusted as a fully unbiased estimate of how well the model will perform in real use.
Key points:
- Rollback decisions based on benchmark scores make the benchmark influence model selection.
- That influence causes overfitting to the benchmark.
- After overfitting, the benchmark is no longer a completely unbiased estimate of performance.
Rubric: The response should state that using benchmark results to choose whether to roll back a model causes overfitting to the benchmark. It should also explain that this overfitting weakens or removes the benchmark's ability to provide an unbiased performance estimate.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main danger of using one holdout set over and over to decide releases, rollbacks, or model changes?
Which dataset should normally be used to compare model versions and decide whether to revert to an earlier checkpoint during development?
What happens if you repeatedly use the test set to choose among model variants?
When is it acceptable to check the test set during model development?
Using the test set to choose whether to keep a model rollback affects the test set's objectivity.
If you keep choosing models by looking at the test set, you begin to _____ the test set.
Classify Each Use of the Test Set as Safe or Unsafe
How Test-Set Feedback Can Make Evaluation Unreliable
Why should the test set stay out of model-tuning decisions?
Using the test set once a week to monitor overall progress is acceptable.
Once the test set has been used to compare models, it can no longer provide a fully _____ estimate of real-world performance.
Match each concept to its meaning when protecting a test set from being used during development.
Order the steps for using a test set without contaminating it during model development.
Why a weekly benchmark should not decide model rollbacks
Spotting the mistake in a model selection decision
What problem can occur if test-set scores are used to decide when to roll back a model version?