Estimating Bias From Training Error and a Human Performance Baseline
Case context: You build a classifier that detects whether satellite photos show flood damage. Its training error is 9%. An experienced human review team, limited by ambiguous images, can do no better than 8% error on the same task.
Question: Given the training error and the best achievable human-level error, what should you conclude about bias, and should bias-reduction be a priority?
Sample answer: The best achievable error is 8%. Because the classifier's training error is 9%, the gap is only 1%. That means the model's avoidable bias appears small, so bias-reduction work should not be the main priority.
Key points:
- The best achievable error is 8%.
- The gap from training error to that level is only 1%.
- The model has little remaining avoidable bias.
- Bias-reduction changes are not the top priority.
Rubric: The learner should identify the 8% baseline, recognize the 1% gap, and conclude that bias-reducing changes should not be prioritized.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Judging Avoidable Bias from the Training Set
A model with 8% training error and 7% best achievable error shows strong evidence of high bias.
Room to Improve When the Best Possible Error Is Near Zero
Matching Bias Clues to Error Gaps
Diagnosing Bias by Comparing Training Error to a Practical Baseline
Why the Same Training Error Can Mean Different Things
Estimating Bias From Training Error and a Human Performance Baseline
What Does a Large Gap Between Training Error and Best Possible Error Suggest?
How to judge whether there is still room to reduce avoidable bias
Can training error by itself tell you whether a bias-reducing change is worthwhile?