Diagnosing a speech command classifier with high training and dev error
Case context: You are building a voice-command classifier that should reach about 4% error. Right now, it has 14% error on the training set and 15% error on the dev set. A teammate suggests collecting 8,000 more labeled recordings before making any other changes.
Question: Using the error rates and target, identify the main issue, assess the proposed fix, and state what should be addressed first to improve the system.
Sample answer: The main issue is avoidable bias, because the training error of 14% is far above the 4% target. Adding 8,000 more recordings is unlikely to solve the problem, since the training and dev errors are already very close, which means variance is not the main limitation. More data can help when the model overfits, but here the bigger need is to improve performance on the current training set. The first step should be to make the learning algorithm fit the training data better before focusing on additional data collection.
Key points:
- Identify avoidable bias as the primary issue because 14% training error is much higher than the 4% goal.
- Explain that collecting more data is not the best first move because it mainly helps with variance, not bias.
- Recommend improving training-set performance first.
Rubric: Diagnose avoidable bias based on the 14% training error versus the 4% target error. Explain why collecting more data is ineffective as a first response because it mainly addresses variance rather than bias. Recommend improving the model’s performance on the training set first.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
An email-spam classifier has 10% training error and 11% development error, but the team wants to reach 3% error. What should they do first?
True or False: When a model has high training error because it does not fit the training set well, collecting more training examples is usually the main fix.
Effect of More Training Examples
Your model’s training error is 18%, and your goal is 6%. What should you address first?
More Data Quickly Solves Underfitting
More training data mainly helps with _____ issues, not with bias.
Match each learning situation to the most appropriate bias-or-variance description.
Order the steps for diagnosing and responding to a model that performs poorly on the training set.
A model has 18% training error, 19% development error, and the target error is 6%. What does this most strongly suggest?
True or False: If training error is still high, adding more data usually leads to big gains on the development and test sets.
If training error remains large, improve the _____ fit before expecting validation or test performance to rise.
Match each error pattern to the correct diagnosis and recommended action.
Prioritize the next improvement step when considering more training data.
Why More Training Data Does Not Fix a Large Training Error
Diagnosing a speech command classifier with high training and dev error
What Should Improve First Before Validation Results Are Expected to Move?