Use Human-Strong Subsets to Keep Improving
A model may already outperform the average human across an entire dev or test set, but human comparison can still be valuable on specific slices where people remain better and the model still makes mistakes. On those slices, human judgments can still supply stronger labels, practical insight into failure modes, and a realistic performance goal.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Human Labeling Works Best for Tasks People Can Do Reliably
Using Human Judgment in Error Review
Use Human-Strong Subsets to Keep Improving
Which set of advantages best explains why comparing with human performance can help ML work?
Comparing a model with strong human performance can help estimate the lowest achievable error and set a realistic target for improvement.
A realistic target error rate can speed up a team’s ____.
Match each reason for comparing against human performance with its use in machine learning work.
Order the reasoning steps for using a human benchmark in model development.
Why comparing algorithm performance to expert performance helps ML teams improve
What to do when a model trails expert performance by a wide margin
Why does a large gap from human performance matter in model debugging?
What choice would most help a team move quickly during model development?
Human-level comparison is useful only for collecting labels.
Learn After
Why Human Comparison Becomes Less Helpful After Strong Model Performance
When can human comparison still be useful after a model is already better than average human performance on the dev or test set?
Human judgment can still be useful on specific subsets even after a model beats average human performance overall.
When people still do better than the model on certain cases, they can provide better _____, useful intuition, and a target level of performance.
Match each benefit of comparing a model against people on a hard subset to its description.
Deciding What to Do After Beating Average Human Performance
In the document-reading example, on which task does the system beat people while people still do better on a different task?
If a classifier beats the average human score on the development set, then comparing its mistakes with human judgments is no longer useful for finding improvements.
Human-comparison methods remain useful as long as there are dev examples where people are _____ and the model is mistaken.
Match each item from the invoice-reading example to its role in the human-better-subset idea.
Order the steps for using a human-strong subset to improve a model.
How human comparison can still help a stronger-than-average system
Using a human-strong subset to improve a transaction review model
When Human-Versus-Model Comparisons Stop Helping