Set the performance target using the strongest human benchmark, not a weaker individual baseline.
Scenario: A plant-quality team is building a system to detect faulty circuit boards. Four human reference points are available: 18% error for a new trainee, 11% for a line operator with moderate experience, 7% for a senior inspector, and 3% for a two-person review team that examines each board together. Management suggests that 7% should be the goal.
Question: Identify the problem with the proposed goal and state the human-level proxy and target error rate that the evidence supports.
Sample answer: The 7% goal is based on the senior individual inspector, but the strongest available human comparison is the two-person review team at 3% error. That means 7% is too loose if the team can already do better. The project should use 3% as the human-level proxy and as the practical target for the system, while recognizing that the true achievable minimum could be even lower.
Key points:
- The review team performs better than the best single inspector.
- A 7% target is worse than an already demonstrated 3% result.
- Use 3% as the human-level proxy and working performance target.
- The true optimal error rate may be below 3%.
Rubric: The response should reject 7% by citing the two-person team, recommend 3% for both the proxy and the desired target, and avoid claiming that 3% must equal the true minimum error.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Which human benchmark should define the proxy for optimal error in a satellite-damage inspection task?
True or false: The best achievable error rate cannot be higher than the 2% error achieved by the specialist panel.
The best available human benchmark is about _____ error.
Match each clinical baseline to its observed error rate.
Order the steps for choosing a human-performance reference point.
Selecting the Human Benchmark for a Performance Target
Set the performance target using the strongest human benchmark, not a weaker individual baseline.
How can a 2% measured error rate constrain the best possible error rate?
What does a specialist team’s 2% error rate support?
If an estimated target error rate is 2%, the true minimum possible error must also be 2%.