Explain the consequences of using test set evaluation scores to make algorithmic rollback decisions.
Question: A machine learning team evaluates their system on the test set once per week to track progress. Explain why using the weekly test set evaluation to decide whether to roll back to a previous version of the algorithm is problematic, and describe the resulting consequences on performance estimation.
Sample answer: Using the test set to make decisions regarding the algorithm, such as whether to roll back to a previous system, causes the algorithm to overfit to the test set. Because of this overfitting, the test set can no longer be counted on to provide a completely unbiased estimate of the system's performance.
Key points:
- Making rollback decisions is an algorithmic decision guided by the test set.
- Using the test set for algorithm decisions leads to overfitting to the test set.
- Overfitting to the test set destroys its ability to provide an unbiased estimate of system performance.
Rubric: The response should clearly identify that making rollback decisions based on test set performance leads to overfitting to the test set. It must also explain that this overfitting prevents the test set from giving a completely unbiased estimate of the system's performance.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Related
What is the primary risk of using the test set to make algorithm decisions, such as whether to roll back to a previous system version?
It is acceptable to use the test set to decide whether to roll back your ML system to a previous version, provided you do so infrequently.
According to Andrew Ng, if you use the test set to make algorithm decisions, you will start to _____ to the test set, rendering it unable to give a completely unbiased estimate of system performance.
Which use of the test set is acceptable when tracking a team's ML progress?
Using the test set to decide whether to roll back to a previous system will cause overfitting to the test set.
When algorithm decisions are based on test set results, the team will start to _____ to the test set.
Match each test set usage to its correct classification as acceptable or problematic.
Order the steps showing how repeated algorithm decisions based on test set scores lead to an unreliable performance estimate.
Why does Machine Learning Yearning emphasize keeping the test set free from algorithm decisions?
Evaluating your system on the test set once per week to track progress is an acceptable practice.
After using the test set for algorithm decisions, it can no longer give a completely _____ estimate of system performance.
Match each concept to its definition as used in the context of test set integrity.
Order the recommended steps for correctly using the test set while maintaining its integrity during ML development.
Explain the consequences of using test set evaluation scores to make algorithmic rollback decisions.
Diagnose the methodological error in a team's test set evaluation process.
What is the risk of using test set performance to guide algorithm rollback decisions?