Why Human-Friendly Tasks Often Produce Better Training Labels
Question: Explain why it is usually easier to collect accurate labels for a machine learning task when most people can already perform the task reliably. Use at least two examples.
Sample answer: When a task is already easy for humans, people can label examples with little special training and usually make fewer mistakes. That means the training set contains cleaner ground truth, which is valuable for supervised learning. For example, many people can correctly label whether an image shows a bicycle or a traffic light. In a more specialized setting, a trained radiologist can label chest X-rays for a particular finding with much higher consistency than a nonexpert. In both cases, human skill makes it possible to create better labels.
Key points:
- If humans can do the task well, they can usually label examples accurately.
- Accurate labels improve the quality of the training data.
- Everyday visual tasks are often easy for general labelers.
- Specialized medical tasks can still be labeled well by domain experts.
Rubric: The response should explain the connection between human task ability and label quality, and include at least two appropriate examples.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Route Difficult Cases to Specialists
Why are labeled examples often easier to collect for tasks people can do accurately?
When people can reliably do a task, trained annotators can usually produce labels accurate enough to supervise a model for that task.
When people can already recognize _____ images easily, human labelers can usually assign accurate labels with little difficulty.
Match each idea to the description that fits human labeling of tasks people can do well.
Put the steps in order for deciding whether people should label the data for a machine learning task.
What error rate can an experienced team of specialists achieve when creating labels for a familiar task?
Human labelers make data easier to annotate only for image classification tasks.
In a chest X-ray labeling example, a team of _____ can provide labels at about a 2% error rate.
Match each labeling case to the reason it can be labeled accurately by people.
Order the reasoning steps that explain why people can serve as effective labelers for tasks humans already do well.
Why Human-Friendly Tasks Often Produce Better Training Labels
Choosing Human Labelers for a Medical Image Project
How Human Skill Affects Label Collection