Learn Before
Sizing a Manual Review Set for a High-Accuracy Defect Detector
Case context: A factory inspection team has improved its defect detector so that the overall error rate is now 1.5%. The lead engineer wants a manual review set with about 60 misclassified product images so the team can study fresh failure patterns.
Question: How many product images should you include in the manual review set to expect about 60 errors, and why?
Sample answer: You should include about 4,000 product images. Since the error rate is 1.5%, or 0.015, you divide the desired number of misclassified examples, 60, by 0.015. That gives 4,000. The calculation shows that when a system is already fairly accurate, you need a much larger review set to collect enough mistakes for analysis.
Key points:
- Divide the target number of errors by the error rate: 60 / 0.015 = 4,000
- Aim for roughly 60 misclassified examples to inspect manually
- Explain that a low error rate requires a larger review set to gather enough mistakes
Rubric: Give full credit if the learner correctly computes 4,000 examples and explains that dividing the desired number of misclassified cases by the 1.5% error rate leads to a much larger dataset.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
If a fraud detector has a 3% error rate, about how large should the eyeball development set be to expect roughly 90 mistakes?
If a classifier makes fewer mistakes, the eyeball dev set can usually be smaller and still collect enough errors for analysis.
A validation set with a 5% mistake rate needs about _____ examples to contain roughly 100 wrong predictions.
Match each classifier-error concept to its dev-set implication.
Order the steps for estimating the required size of an Eyeball dev set from a model's error rate.
Why does a low classifier error rate call for a larger review set?
A classifier with an 8% error rate needs an Eyeball dev set of 2,500 examples to collect about 100 mistakes.
The _____ the classifier error rate, the larger the review set must be to collect enough misclassified examples.
Approximate Review-Set Sizes Needed to See About 100 Errors
Order the logic that explains why a better classifier can require a bigger dev set for error review.
How Dev-Set Error Rate Affects Eyeball Dev Set Size
Sizing a Manual Review Set for a High-Accuracy Defect Detector
Why a 2,000-Item Manual Review Set Fits a 5% Error Rate