Selecting the Human Benchmark for a Performance Target
Question: In a concise analytical response, explain why a 2% miss rate from the specialist review panel is a better basis than 5% or 10% when estimating the human benchmark and setting a target level.
Sample answer: The panel's 2% miss rate is the strongest human result in the comparison set, so it is the most trustworthy estimate of what people can currently achieve. The 5% and 10% figures are not suitable benchmarks because a better human team has already reached 2%. For that reason, 2% is a sensible stand-in for both the estimated benchmark and the desired target. The actual ideal could still be lower than 2%, but it cannot be higher if the panel has already shown 2% performance.
Key points:
- 2% is the best demonstrated human result.
- 5% and 10% are weaker than the available human baseline.
- 2% can serve as both the benchmark estimate and the target.
- The true ideal may be below 2%, but not above it.
Rubric:
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Which human benchmark should define the proxy for optimal error in a satellite-damage inspection task?
True or false: The best achievable error rate cannot be higher than the 2% error achieved by the specialist panel.
The best available human benchmark is about _____ error.
Match each clinical baseline to its observed error rate.
Order the steps for choosing a human-performance reference point.
Selecting the Human Benchmark for a Performance Target
Set the performance target using the strongest human benchmark, not a weaker individual baseline.
How can a 2% measured error rate constrain the best possible error rate?
What does a specialist team’s 2% error rate support?
If an estimated target error rate is 2%, the true minimum possible error must also be 2%.