Learn Before
When Human-Versus-Model Comparisons Stop Helping
Question: In the human-better subset idea, at what point do comparisons against human performance stop being useful for improving a machine learning system?
Sample answer: They stop only when the dev set no longer contains any cases where a person is correct and the model is wrong. Surpassing the average human score overall is not enough by itself.
Key points:
- The method still works as long as such cases remain.
- What matters is the absence of any human-correct, model-wrong examples.
- An above-average overall score does not eliminate those subsets.
Rubric: Give credit only if the answer says the technique ends when there are no remaining examples in which humans outperform the model.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why Human Comparison Becomes Less Helpful After Strong Model Performance
When can human comparison still be useful after a model is already better than average human performance on the dev or test set?
Human judgment can still be useful on specific subsets even after a model beats average human performance overall.
When people still do better than the model on certain cases, they can provide better _____, useful intuition, and a target level of performance.
Match each benefit of comparing a model against people on a hard subset to its description.
Deciding What to Do After Beating Average Human Performance
In the document-reading example, on which task does the system beat people while people still do better on a different task?
If a classifier beats the average human score on the development set, then comparing its mistakes with human judgments is no longer useful for finding improvements.
Human-comparison methods remain useful as long as there are dev examples where people are _____ and the model is mistaken.
Match each item from the invoice-reading example to its role in the human-better-subset idea.
Order the steps for using a human-strong subset to improve a model.
How human comparison can still help a stronger-than-average system
Using a human-strong subset to improve a transaction review model
When Human-Versus-Model Comparisons Stop Helping