Analyzing Error Metrics for Data Mismatch
Case context: You are developing an ML system and have tracked its performance across different data splits. You observe a 10% error on the training set, an 11% error on the training-dev set, and a 20% error on the dev set.
Question: Based on these metrics, what should you diagnose as the primary causes of the model's errors, and what issue can you confidently rule out?
Sample answer: You should diagnose the system as suffering from high avoidable bias (indicated by the 10% training error) and data mismatch (indicated by the jump from 11% training-dev error to 20% dev error). You can rule out high variance on the training set distribution because the training-dev error is only 1% higher than the training error.
Key points:
- Diagnose high avoidable bias.
- Diagnose data mismatch.
- Rule out high variance on the training-set distribution.
Rubric: A correct diagnosis will explicitly state that the model suffers from high avoidable bias and data mismatch, while correctly ruling out high variance on the training set distribution based on the provided error rates.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What does the 1% gap between training error (10%) and training-dev error (11%) indicate in the ML Yearning scenario?
True or False: An algorithm with 10% training error, 11% training-dev error, and 20% dev error suffers from high variance on the training set distribution.
With 10% training error, 11% training-dev error, and 20% dev error, the algorithm suffers from high avoidable bias and _____, but not high variance.
Match each error gap from the 10%/11%/20% scenario to the ML problem it diagnoses.
Order the diagnostic reasoning steps used to conclude the 10%/11%/20% algorithm has avoidable bias and data mismatch but not high variance.
Which combination of problems does an algorithm with 10% training error, 11% training-dev error, and 20% dev error have (assuming ~0% human-level error)?
True or False: In the 10%/11%/20% scenario, data mismatch contributes a larger performance drop than variance when going from training to dev error.
In the 10%/11%/20% scenario, the gap between training error and _____ error is used to assess variance on the training set distribution.
Match each ML problem type to the error evidence that confirms or rules it out in the 10%/11%/20% scenario.
Order the three error measurements from lowest to highest as reported in ML Yearning's high-avoidable-bias and data-mismatch scenario.
Diagnose the 10%/11%/20% error scenario
Analyzing Error Metrics for Data Mismatch
Identify the missing error problem