Learn Before
The Best Human Benchmark Depends on How Accurate the System Already Is
When a model is still making a large number of mistakes, small differences between human references may not change the improvement plan much. For instance, if a classifier has 38% error, using a clinician with 11% error instead of one with 4% error may not affect the next debugging step very much. But if the model is already near 9% error, comparing against a 2% human reference gives a much clearer target for further gains.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Using Experience to Set a Target Error Rate
The Best Human Benchmark Depends on How Accurate the System Already Is
What does the term desired error rate mean?
True or False: A performance target can be drawn on a learning curve.
A target error rate is the level of error a model is hoped to eventually _____ .
Match each term to its role in target-error analysis.
Arrange the steps for adding a target error level to a model’s learning curve.
On which plot should a target performance level be marked?
True or False: A desired error rate is the level of error a model has already reached on the development set.
The target performance should be added to your _____.
Match each element on a performance plot to its meaning.
Put the desired error rate workflow in order
Using a Target Error Rate on a Learning Curve
Add a target line to a learning curve for a churn-prediction project.
Adding a target performance level to a learning plot
Learn After
How Human Performance Guides Machine Learning Work
At 28% system error, which benchmark is most useful for deciding where to improve next?
If a classifier has about 40% error, using a nurse practitioner with 12% error instead of a senior specialist with 6% error as the human reference makes only a small practical difference.
Using a human benchmark when error is already fairly low
Match each model error situation to the lesson it gives about choosing a human benchmark.
Order the steps for deciding when a more precise human benchmark is worth using.
Why is a 2% human benchmark more useful for guiding improvement when a system has 10% error than when it has 40% error?
At a 40% system error rate, switching between a 12% human benchmark and a 6% human benchmark usually changes the diagnosis a great deal.
When a model is meant to match expert inspectors, their error rate can be the _____ error rate for the system.
Match each labeler or system case to its description.
Order the steps for choosing a human performance reference when evaluating an ML system.
How should the choice of human reference change as a model gets better?
Select the right human reference when system error is very high.
Why a tighter human reference helps when error is already low