Evaluating a Sales Forecast Model
Situation: A team is checking a model that predicts weekly store demand. Their target error rate is 5%. On the training set, the model’s error has leveled off at 18%. On the development set, the error is 31%.
Question: What do these results suggest about the model’s bias and variance, and what should the team try to do next?
Sample answer: The model shows high bias because its training error is still far above the target level. It also shows high variance because the development error is much worse than the training error. The next step is to work on reducing both bias and variance.
Key points:
- High bias is indicated by 18% training error compared with a 5% target.
- High variance is indicated by 31% development error compared with 18% training error.
- The recommended direction is to reduce both sources of error.
Rubric: The response must correctly identify high bias from the large training error, identify high variance from the training-versus-development gap, and recommend reducing both.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Recognizing When Both Bias and Variance Are High
If a model has both high bias and high variance, its training error and development error are usually about the same.
Learning Curves That Point to _____
Reading Training and Validation Patterns
Diagnosing Bias and Variance in a Model
What a learning curve suggests when both underfitting and overfitting are present
Evaluating a Sales Forecast Model
What does a large validation gap indicate when training error is already high?
What should you do when the learning curve suggests both weak fit and high instability?
When a classifier has high bias, it usually achieves very low training error.