Learn Before
Choosing which error categories to fix first
Case context: A team reviews 150 incorrect predictions from a document classifier. Their error log shows that "Name mismatch" accounts for 6% of the mistakes, while "Low scan quality" and "Unusual formatting" account for most of the rest.
Question: Based on standard error-analysis guidance, what should the team choose to investigate or improve next, and why?
Sample answer: The team should focus on "Low scan quality" or "Unusual formatting" rather than "Name mismatch." Fixing the name-mismatch cases could improve performance by at most 6% of the total errors, so it offers a smaller payoff than the more common categories.
Key points:
- Prioritize the more frequent categories such as Low scan quality or Unusual formatting.
- Fixing Name mismatch can remove at most 6% of the errors.
- Higher-frequency error types usually offer a larger opportunity for improvement.
Rubric: The response must say that the team should prioritize "Low scan quality" or "Unusual formatting" over "Name mismatch," and it must justify that choice by noting that correcting "Name mismatch" can improve results by no more than 6%.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
After reviewing 150 misclassified drone-delivery images, package-label errors account for 9% of the mistakes, weather-haze errors account for 18%, and shadow-occlusion errors account for 22%. Which category should you likely prioritize?
If 120 validation errors are reviewed and 9 of them are bicycle mistakes, then fixing bicycle errors perfectly could remove at most 7.5% of the total errors.
Error analysis on 100 misclassified dev set examples shows that traffic-light mistakes can eliminate _____ of the errors at most.
Match each error group to the statement that best describes why it deserves attention.
Put the steps in order for using error-category counts to decide where to focus next.
What does an error category's share of dev-set mistakes tell you?
Should the easiest error category always get top priority in error analysis?
Error-category frequency gives an upper bound on improvement.
Match each error-analysis term with its meaning.
Put the analysis steps in the order that justifies not prioritizing one low-frequency error type.
Explain How Error Category Frequency Guides Project Prioritization
Choosing which error categories to fix first
Greatest Error Reduction From Fixing One Category