Route Difficult Cases to Specialists

Why are labeled examples often easier to collect for tasks people can do accurately?

When people can reliably do a task, trained annotators can usually produce labels accurate enough to supervise a model for that task.

When people can already recognize _____ images easily, human labelers can usually assign accurate labels with little difficulty.

Match each idea to the description that fits human labeling of tasks people can do well.

Put the steps in order for deciding whether people should label the data for a machine learning task.

What error rate can an experienced team of specialists achieve when creating labels for a familiar task?

Human labelers make data easier to annotate only for image classification tasks.

In a chest X-ray labeling example, a team of _____ can provide labels at about a 2% error rate.

Match each labeling case to the reason it can be labeled accurately by people.

Order the reasoning steps that explain why people can serve as effective labelers for tasks humans already do well.

Why Human-Friendly Tasks Often Produce Better Training Labels

Choosing Human Labelers for a Medical Image Project

How Human Skill Affects Label Collection