Applying Human Labeling to a Medical AI Initiative
Case context: You are leading an AI team developing a machine learning model to detect anomalies in X-ray images. Your team is deciding how to acquire a large, accurate dataset to train the algorithm, as current automated heuristics are failing to produce reliable labels.
Question: Given that interpreting X-rays is a task performed well by trained professionals, what labeling strategy should you decide to implement, and what error rate might you realistically target based on Machine Learning Yearning principles?
Sample answer: I would decide to use a team of doctors as human labelers to annotate the X-ray images. Because medical professionals already perform this task well, they can provide high-accuracy labels. Based on Andrew Ng's examples, it is realistic to expect a team of doctors to provide these labels with a low error rate, such as around 2%.
Key points:
- Employ human experts (doctors) as labelers.
- Medical imaging is a task humans (doctors) perform well.
- Human experts provide high-accuracy labels.
- A team of doctors can achieve a low error rate (e.g., 2%).
Rubric: The learner must recommend using human experts (doctors) to label the data and mention that humans can achieve a low error rate (specifically citing the ~2% rate for medical imaging teams).
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Escalating Hard Labels to Experts
Why is obtaining labeled data easier for tasks that humans perform well?
For tasks that humans perform well, human labelers can provide high-accuracy labels to train ML algorithms.
Andrew Ng writes that since people recognize _____ images well, it is straightforward for human labelers to provide high-accuracy labels.
Match each concept to its correct description regarding human labeling of human-solvable tasks.
Order the steps for deciding whether human labelers are a good fit for collecting labeled data for an ML task.
In Andrew Ng's medical imaging example, what error rate can a team of doctors achieve when providing labels?
According to Machine Learning Yearning, the ease-of-labeling advantage from human labelers applies only to image recognition tasks.
In Andrew Ng's medical imaging example, a team of _____ can provide labels at a 2% error rate.
Match each labeling scenario to the property that best explains it, based on Machine Learning Yearning.
Order Andrew Ng's reasoning steps for why human-solvable tasks benefit from human labelers.
Analyzing the Labeling Advantage for Human-Solvable Tasks
Applying Human Labeling to a Medical AI Initiative
Impact of Human Proficiency on Data Acquisition