Select the right human reference when system error is very high.
Scenario: A company is building a vision model that sorts package photos into damaged versus undamaged. The current system makes 38% errors. The team can use annotations from floor supervisors with a 12% error rate or from a small group of veteran quality specialists with a 4% error rate.
Question: Which group should the team prioritize for labels and for generating practical intuitions? Explain using the idea of a human-level reference.
Sample answer: The floor supervisors are sufficient. Because the system error is still very high at 38%, the difference between a 12% reference and a 4% reference is too small to matter much at this stage. Either group is far better than the system, and the cheaper reference can still provide useful intuition and labels for improvement.
Key points:
- The system error is large.
- The gap between 12% and 4% is not the main bottleneck here.
- The lower-cost human reference is adequate for guidance.
Rubric: A correct answer chooses the floor supervisors and explains that, with such a high system error rate, the exact human reference error is not critical for the next improvement step.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
How Human Performance Guides Machine Learning Work
At 28% system error, which benchmark is most useful for deciding where to improve next?
If a classifier has about 40% error, using a nurse practitioner with 12% error instead of a senior specialist with 6% error as the human reference makes only a small practical difference.
Using a human benchmark when error is already fairly low
Match each model error situation to the lesson it gives about choosing a human benchmark.
Order the steps for deciding when a more precise human benchmark is worth using.
Why is a 2% human benchmark more useful for guiding improvement when a system has 10% error than when it has 40% error?
At a 40% system error rate, switching between a 12% human benchmark and a 6% human benchmark usually changes the diagnosis a great deal.
When a model is meant to match expert inspectors, their error rate can be the _____ error rate for the system.
Match each labeler or system case to its description.
Order the steps for choosing a human performance reference when evaluating an ML system.
How should the choice of human reference change as a model gets better?
Select the right human reference when system error is very high.
Why a tighter human reference helps when error is already low