Choosing Human Labelers for a Medical Image Project
Case context: You are leading a machine learning project that must classify findings in dermatology images. Automated heuristics are producing inconsistent labels, and the team needs a reliable way to create a large training set. Interpreting these images is a task that trained specialists already do well.
Question: What labeling strategy should you choose, and what level of labeling error is realistic under these conditions?
Sample answer: The best choice is to use human experts, such as dermatologists, to label the images. Because specialists already perform this kind of interpretation well, they can produce accurate labels. For a team of experts working on a medical imaging task like this, a low error rate around 2% is a realistic target.
Key points:
- Use expert human labelers
- Pick a task that trained people already do well
- Expect high-quality labels from specialists
- A team of experts can often reach a low error rate, such as about 2%
Rubric: The learner must recommend using human experts to label the data and note that a low error rate, roughly 2%, is realistic for a medical imaging team.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Route Difficult Cases to Specialists
Why are labeled examples often easier to collect for tasks people can do accurately?
When people can reliably do a task, trained annotators can usually produce labels accurate enough to supervise a model for that task.
When people can already recognize _____ images easily, human labelers can usually assign accurate labels with little difficulty.
Match each idea to the description that fits human labeling of tasks people can do well.
Put the steps in order for deciding whether people should label the data for a machine learning task.
What error rate can an experienced team of specialists achieve when creating labels for a familiar task?
Human labelers make data easier to annotate only for image classification tasks.
In a chest X-ray labeling example, a team of _____ can provide labels at about a 2% error rate.
Match each labeling case to the reason it can be labeled accurately by people.
Order the reasoning steps that explain why people can serve as effective labelers for tasks humans already do well.
Why Human-Friendly Tasks Often Produce Better Training Labels
Choosing Human Labelers for a Medical Image Project
How Human Skill Affects Label Collection