Learn Before
How human comparison can still help a stronger-than-average system
Question: A speech recognition model now performs better than the average transcriber on the full development and test sets. Explain how human comparison can still be useful for improving it. State the condition that must hold and describe the benefits human judgment provides.
Sample answer: Human comparison can still be valuable if there is a slice of data where people still do better than the model. On that slice, human labels can be higher quality than the model's predictions, and the human decisions can help the team understand what the system is missing. Those human results also give a realistic target for that difficult slice, even if the overall system is already stronger than the average person.
Key points:
- There must be a subset where humans outperform the model.
- Human labels on that subset can be more accurate.
- Human judgment can explain why the model failed on those cases.
- Human performance on the subset provides a goal for improvement.
Rubric: Full credit requires stating that a human-better subset must exist, and naming all three benefits: better labels, useful intuition about failures, and a performance target for that subset.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why Human Comparison Becomes Less Helpful After Strong Model Performance
When can human comparison still be useful after a model is already better than average human performance on the dev or test set?
Human judgment can still be useful on specific subsets even after a model beats average human performance overall.
When people still do better than the model on certain cases, they can provide better _____, useful intuition, and a target level of performance.
Match each benefit of comparing a model against people on a hard subset to its description.
Deciding What to Do After Beating Average Human Performance
In the document-reading example, on which task does the system beat people while people still do better on a different task?
If a classifier beats the average human score on the development set, then comparing its mistakes with human judgments is no longer useful for finding improvements.
Human-comparison methods remain useful as long as there are dev examples where people are _____ and the model is mistaken.
Match each item from the invoice-reading example to its role in the human-better-subset idea.
Order the steps for using a human-strong subset to improve a model.
How human comparison can still help a stronger-than-average system
Using a human-strong subset to improve a transaction review model
When Human-Versus-Model Comparisons Stop Helping