How Human Skill Affects Label Collection
Question: In a task that people can do reliably, how does that affect the process of creating labeled training data?
Sample answer: When people can perform the task accurately, labels are easier to collect because human annotators can produce dependable ground-truth examples with relatively little error. That reduces the difficulty of building the dataset.
Key points:
- Tasks that humans can do well are easier to label.
- Human annotators can produce more accurate labels.
- Reliable labels are easier to obtain when the task is straightforward for people.
Rubric: A correct response should explain that strong human performance makes labeling easier by allowing annotators to generate accurate ground-truth labels.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Route Difficult Cases to Specialists
Why are labeled examples often easier to collect for tasks people can do accurately?
When people can reliably do a task, trained annotators can usually produce labels accurate enough to supervise a model for that task.
When people can already recognize _____ images easily, human labelers can usually assign accurate labels with little difficulty.
Match each idea to the description that fits human labeling of tasks people can do well.
Put the steps in order for deciding whether people should label the data for a machine learning task.
What error rate can an experienced team of specialists achieve when creating labels for a familiar task?
Human labelers make data easier to annotate only for image classification tasks.
In a chest X-ray labeling example, a team of _____ can provide labels at about a 2% error rate.
Match each labeling case to the reason it can be labeled accurately by people.
Order the reasoning steps that explain why people can serve as effective labelers for tasks humans already do well.
Why Human-Friendly Tasks Often Produce Better Training Labels
Choosing Human Labelers for a Medical Image Project
How Human Skill Affects Label Collection