How Human Performance Guides Machine Learning Work
A strong human benchmark is useful in several ways. It helps supply labeled examples for tasks people can judge reliably, it supports error analysis because humans can often explain patterns a model misses, and it gives a reference point for estimating the best achievable error. That reference helps a team set a realistic target, and if the model is still far from it, the remaining gap suggests there is substantial room to improve.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
How Human Performance Guides Machine Learning Work
At 28% system error, which benchmark is most useful for deciding where to improve next?
If a classifier has about 40% error, using a nurse practitioner with 12% error instead of a senior specialist with 6% error as the human reference makes only a small practical difference.
Using a human benchmark when error is already fairly low
Match each model error situation to the lesson it gives about choosing a human benchmark.
Order the steps for deciding when a more precise human benchmark is worth using.
Why is a 2% human benchmark more useful for guiding improvement when a system has 10% error than when it has 40% error?
At a 40% system error rate, switching between a 12% human benchmark and a 6% human benchmark usually changes the diagnosis a great deal.
When a model is meant to match expert inspectors, their error rate can be the _____ error rate for the system.
Match each labeler or system case to its description.
Order the steps for choosing a human performance reference when evaluating an ML system.
How should the choice of human reference change as a model gets better?
Select the right human reference when system error is very high.
Why a tighter human reference helps when error is already low
Learn After
Human Labeling Works Best for Tasks People Can Do Reliably
Using Human Judgment in Error Review
Use Human-Strong Subsets to Keep Improving
Which set of advantages best explains why comparing with human performance can help ML work?
Comparing a model with strong human performance can help estimate the lowest achievable error and set a realistic target for improvement.
A realistic target error rate can speed up a team’s ____.
Match each reason for comparing against human performance with its use in machine learning work.
Order the reasoning steps for using a human benchmark in model development.
Why comparing algorithm performance to expert performance helps ML teams improve
What to do when a model trails expert performance by a wide margin
Why does a large gap from human performance matter in model debugging?
What choice would most help a team move quickly during model development?
Human-level comparison is useful only for collecting labels.