Human Labeling Works Best for Tasks People Can Do Reliably
When a labeling task is something people can perform accurately, it is usually easier to collect supervised data because annotators can produce dependable labels. For instance, workers can often label clear photos of everyday objects with little difficulty, and specialists can also assign high-quality labels in domains they know well.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Human Labeling Works Best for Tasks People Can Do Reliably
Using Human Judgment in Error Review
Use Human-Strong Subsets to Keep Improving
Which set of advantages best explains why comparing with human performance can help ML work?
Comparing a model with strong human performance can help estimate the lowest achievable error and set a realistic target for improvement.
A realistic target error rate can speed up a team’s ____.
Match each reason for comparing against human performance with its use in machine learning work.
Order the reasoning steps for using a human benchmark in model development.
Why comparing algorithm performance to expert performance helps ML teams improve
What to do when a model trails expert performance by a wide margin
Why does a large gap from human performance matter in model debugging?
What choice would most help a team move quickly during model development?
Human-level comparison is useful only for collecting labels.
Learn After
Route Difficult Cases to Specialists
Why are labeled examples often easier to collect for tasks people can do accurately?
When people can reliably do a task, trained annotators can usually produce labels accurate enough to supervise a model for that task.
When people can already recognize _____ images easily, human labelers can usually assign accurate labels with little difficulty.
Match each idea to the description that fits human labeling of tasks people can do well.
Put the steps in order for deciding whether people should label the data for a machine learning task.
What error rate can an experienced team of specialists achieve when creating labels for a familiar task?
Human labelers make data easier to annotate only for image classification tasks.
In a chest X-ray labeling example, a team of _____ can provide labels at about a 2% error rate.
Match each labeling case to the reason it can be labeled accurately by people.
Order the reasoning steps that explain why people can serve as effective labelers for tasks humans already do well.
Why Human-Friendly Tasks Often Produce Better Training Labels
Choosing Human Labelers for a Medical Image Project
How Human Skill Affects Label Collection