What is the risk of using test set performance to guide algorithm rollback decisions?
Question: If a team uses regular test set evaluation to decide whether to roll back to a previous system version, what is the primary risk to the test set's utility?
Sample answer: The primary risk is that the team will overfit to the test set. When this happens, the test set can no longer provide a completely unbiased estimate of the system's performance.
Key points:
- It leads to overfitting to the test set.
- The test set can no longer give a completely unbiased estimate of system performance.
Rubric: The answer should state that the primary risk is overfitting to the test set, which prevents it from being a completely unbiased estimate of performance.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Related
What is the primary risk of using the test set to make algorithm decisions, such as whether to roll back to a previous system version?
It is acceptable to use the test set to decide whether to roll back your ML system to a previous version, provided you do so infrequently.
According to Andrew Ng, if you use the test set to make algorithm decisions, you will start to _____ to the test set, rendering it unable to give a completely unbiased estimate of system performance.
Which use of the test set is acceptable when tracking a team's ML progress?
Using the test set to decide whether to roll back to a previous system will cause overfitting to the test set.
When algorithm decisions are based on test set results, the team will start to _____ to the test set.
Match each test set usage to its correct classification as acceptable or problematic.
Order the steps showing how repeated algorithm decisions based on test set scores lead to an unreliable performance estimate.
Why does Machine Learning Yearning emphasize keeping the test set free from algorithm decisions?
Evaluating your system on the test set once per week to track progress is an acceptable practice.
After using the test set for algorithm decisions, it can no longer give a completely _____ estimate of system performance.
Match each concept to its definition as used in the context of test set integrity.
Order the recommended steps for correctly using the test set while maintaining its integrity during ML development.
Explain the consequences of using test set evaluation scores to make algorithmic rollback decisions.
Diagnose the methodological error in a team's test set evaluation process.
What is the risk of using test set performance to guide algorithm rollback decisions?