Learn Before
Using a human-strong subset to improve a transaction review model
Case context: Your model for flagging suspicious insurance claims performs better than human reviewers overall on the development and test data. But an error review shows that on claims from a newly opened region, the human adjusters are still more accurate than the model.
Question: How can you use that regional subset to keep improving the system, even though its average performance is already better than humans overall?
Sample answer: Since people still beat the model on the regional subset, that slice is useful for human comparison. We can ask experienced adjusters for cleaner labels on those claims, study their judgment to understand why they are correct when the model fails, and use their accuracy on that subset as the bar the system should try to match.
Key points:
- Treat the regional subset as a place where human comparison still applies.
- Collect better labels from human reviewers for those claims.
- Use human judgment to analyze model mistakes on that slice.
- Use human accuracy on that subset as the target to beat.
Rubric: Checks whether the response applies the three human-comparison uses—labels, intuition, and targets—to the regional claims subset.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why Human Comparison Becomes Less Helpful After Strong Model Performance
When can human comparison still be useful after a model is already better than average human performance on the dev or test set?
Human judgment can still be useful on specific subsets even after a model beats average human performance overall.
When people still do better than the model on certain cases, they can provide better _____, useful intuition, and a target level of performance.
Match each benefit of comparing a model against people on a hard subset to its description.
Deciding What to Do After Beating Average Human Performance
In the document-reading example, on which task does the system beat people while people still do better on a different task?
If a classifier beats the average human score on the development set, then comparing its mistakes with human judgments is no longer useful for finding improvements.
Human-comparison methods remain useful as long as there are dev examples where people are _____ and the model is mistaken.
Match each item from the invoice-reading example to its role in the human-better-subset idea.
Order the steps for using a human-strong subset to improve a model.
How human comparison can still help a stronger-than-average system
Using a human-strong subset to improve a transaction review model
When Human-Versus-Model Comparisons Stop Helping