Case study: Designing an error analysis spreadsheet for a dog classifier with incorrect labels.
Case context: A machine learning developer is building a dog and great cat classifier. During error analysis, they suspect that a noticeable portion of the validation set images have incorrect ground-truth labels, which might be skewing the error rates. They want to check if the volume of these errors is large enough to warrant a complete dataset cleaning.
Question: Based on the text, what action should the developer take in their error analysis spreadsheet to investigate this, and what specific columns should their spreadsheet contain?
Sample answer: The developer should add a 'Mislabeled' category to the error analysis spreadsheet to track the fraction of examples that are mislabeled. The spreadsheet should be structured with the following six columns: Image, Dog, Great cat, Blurry, Mislabeled, and Comments.
Key points:
- Add a 'Mislabeled' category to track the fraction of examples that are mislabeled.
- Use the specific columns: Image, Dog, Great cat, Blurry, Mislabeled, and Comments.
Rubric: A correct response must identify that a 'Mislabeled' category should be added to keep track of the fraction of mislabeled examples. Additionally, it must list the six columns: Image, Dog, Great cat, Blurry, Mislabeled, and Comments.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
When should you add a 'Mislabeled' category to your error analysis spreadsheet?
True or False: You should add a 'Mislabeled' category to your error analysis spreadsheet only when you are certain—not merely suspicious—that examples are mislabeled.
When you suspect a significant fraction of dev set examples have incorrect labels, you should add a _____ column to your error analysis spreadsheet.
When should you add a 'Mislabeled' category to your error analysis spreadsheet according to Machine Learning Yearning?
The 'Mislabeled' column in the error analysis spreadsheet tracks examples where the ground-truth label itself is incorrect.
You should add a Mislabeled category to the error analysis spreadsheet when you suspect the _____ of mislabeled images is significant.
Match each column from the Machine Learning Yearning error analysis spreadsheet to what it captures.
Order the steps for setting up and using a Mislabeled column in an error analysis spreadsheet.
Which of the following is NOT listed as a column in the error analysis spreadsheet shown in Machine Learning Yearning (p. 33)?
According to Machine Learning Yearning, a 'Mislabeled' column should be included in every error analysis spreadsheet as a standard default.
The error analysis spreadsheet in Machine Learning Yearning has columns: Image, Dog, Great cat, Blurry, _____, and Comments.
Match each description to the corresponding column name in the Machine Learning Yearning error analysis spreadsheet.
Order the reasoning steps that lead a practitioner to add a 'Mislabeled' column rather than ignoring suspected label errors.
Explain how to set up error analysis when dataset labels are suspected to be incorrect.
Case study: Designing an error analysis spreadsheet for a dog classifier with incorrect labels.
Identify the tracking purpose and trigger for the 'Mislabeled' column.