Using Human Judgment in Error Review
When a model still performs worse than people on a task, error review can use human judgment to identify which clues led a person to the correct answer. That insight can then guide changes to the data, features, or model design. In medical imaging, talking through difficult cases with clinicians can reveal the cues they rely on and suggest useful improvements.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Human Labeling Works Best for Tasks People Can Do Reliably
Using Human Judgment in Error Review
Use Human-Strong Subsets to Keep Improving
Which set of advantages best explains why comparing with human performance can help ML work?
Comparing a model with strong human performance can help estimate the lowest achievable error and set a realistic target for improvement.
A realistic target error rate can speed up a team’s ____.
Match each reason for comparing against human performance with its use in machine learning work.
Order the reasoning steps for using a human benchmark in model development.
Why comparing algorithm performance to expert performance helps ML teams improve
What to do when a model trails expert performance by a wide margin
Why does a large gap from human performance matter in model debugging?
What choice would most help a team move quickly during model development?
Human-level comparison is useful only for collecting labels.
Learn After
When is it especially useful to apply human-style error analysis?
A useful aim of error analysis is to infer which clues a person relies on to reach the correct label.
A transcript turned “blueberry muffins” into “_____ muffins.”
Match each part of expert-guided error analysis to its role.
Arrange the steps for human-guided error analysis on a weak model.
How can domain expertise be used when analyzing errors in chest X-ray classification?
Human-guided error analysis is most informative when the model still performs worse than people.
In error analysis, human _____ can help reveal which clues people use to reach the right label.
Connect each case to the way human judgment helps diagnose model mistakes.
Order the steps for using expert intuition to improve a crop-disease image classifier.
Use human judgment to interpret model mistakes
Using clinician insight to improve a chest scan classifier
Using Human Judgment in Error Analysis