Use the Strongest Practical Human Benchmark as a Proxy
If more than one human reference point is available, it is usually best to take the strongest realistic human or human-team result as the stand-in for near-optimal performance. For example, if a group of physicians makes 2% errors on an imaging task, that 2% rate is a sensible benchmark for the target error rate. A looser target such as 5% or 10% would be too weak, because the human team already does better.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Use the Strongest Practical Human Benchmark as a Proxy
Estimating Human Error on Mobile Cat Images
A Large Training-to-Validation Gap Signals High Variance
Why is the optimal error rate for detecting a bicycle in a clear photo close to 0%?
If 8% of satellite images are impossible for experts to label correctly because thick cloud cover hides the target, the best achievable classification error is approximately 8%.
Human-level performance is often used as a proxy for the _____ error rate on a task.
Match each situation with the approximate best possible error rate suggested by human performance.
Order the steps for using human performance as a stand-in for the best achievable error rate and deciding whether to reduce bias.
A defect detector has 12% error, while expert annotators have 4% error. What is the avoidable bias and what should you do next?
Assuming a Zero Error Floor
In a task where people almost never miss the correct answer, the ideal error rate is nearly _____.
Match each term to its meaning when human performance is used as a practical benchmark for model error.
Order the steps for deciding whether a task has a near-zero or a much higher best-possible error rate.
How model and expert error comparison reveals bias
Selecting a Desired Error Rate Proxy for a Noisy Dictation System
What information should a person use when checking one stage of a pipeline against human performance?
Learn After
Which human benchmark should define the proxy for optimal error in a satellite-damage inspection task?
True or false: The best achievable error rate cannot be higher than the 2% error achieved by the specialist panel.
The best available human benchmark is about _____ error.
Match each clinical baseline to its observed error rate.
Order the steps for choosing a human-performance reference point.
Selecting the Human Benchmark for a Performance Target
Set the performance target using the strongest human benchmark, not a weaker individual baseline.
How can a 2% measured error rate constrain the best possible error rate?
What does a specialist team’s 2% error rate support?
If an estimated target error rate is 2%, the true minimum possible error must also be 2%.