Learn Before
Determine the labeling workflow for a new medical imaging dataset.
Case context: A machine learning team is building a dataset of medical images. They have access to one junior doctor and a panel of three highly experienced senior doctors, whose time is very expensive.
Question: Based on Andrew Ng's advice, how should the team assign the labeling tasks among the junior doctor and the senior doctors to balance quality and cost?
Sample answer: The team should have the single junior doctor review and label the vast majority of the medical images. If the junior doctor encounters particularly difficult or ambiguous cases, those specific images should be brought to the panel of senior doctors for discussion and labeling.
Key points:
- Assign the vast majority of standard cases to the junior doctor.
- Identify hard cases during the initial junior doctor review.
- Escalate only the hard cases to the expensive senior doctors.
Rubric: The answer must clearly propose assigning the bulk of the labeling to the junior doctor and reserving the panel of senior doctors exclusively for the hard cases.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why does Andrew Ng advise against routing every labeled example to a full team of expert doctors?
True or False: In the escalation labeling strategy, the junior expert is responsible for labeling every case, including the hardest ones.
In the escalation labeling strategy, a single _____ expert labels most cases while harder ones are sent to more experienced experts.
Match each type of labeling case or resource to its role in the escalation labeling strategy.
Order the steps for implementing a cost-efficient tiered labeling workflow using expert escalation.
In the escalation labeling strategy, what is the correct action when a junior expert encounters a case too difficult to label confidently?
True or False: The escalation labeling strategy requires every example in the dataset to be reviewed by at least one senior expert.
Andrew Ng suggests having a single junior doctor label the _____ of cases and escalating only the harder ones to more experienced doctors.
Match each key concept from the escalation labeling strategy to its correct description.
Order the reasoning steps Andrew Ng uses to justify a tiered, cost-efficient labeling approach.
Analyze the cost-efficiency of escalating hard labels in data annotation.
Determine the labeling workflow for a new medical imaging dataset.
Identify the primary motivation for using an escalation labeling strategy.