Google

A concrete way to carry out error analysis is to manually examine about 100 dev set examples that the algorithm misclassifies and count the major categories of errors. This information can be used to prioritize what types of errors to work on fixing.

Manually Reviewing 100 Speech Recognition Dev Set Examples

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

What is the primary purpose of manual error analysis?

Should error analysis focus on correctly classified examples?

Manual error analysis is carried out on about 100 misclassified _____ set examples.

Match components of manual error analysis to their descriptions.

Order the steps of manual error analysis.

Question:
Explain how manually reviewing about 100 misclassified dev set examples and counting error categories helps a machine learning team prioritize what types of errors to work on fixing.

Sample answer:
Manually reviewing a sample of 100 misclassified dev set examples allows a team to classify the errors into distinct categories. By counting the fraction of errors in each category, the team can estimate the maximum impact of fixing each category. This quantitative feedback helps prioritize engineering work on the categories that cause the most errors.

Key points:
- Gather a sample of about 100 misclassified dev set examples
- Manually inspect and classify the errors into categories
- Count the frequency of each major error category
- Use the counts to prioritize engineering tasks on high-impact categories

Rubric:
The response must describe gathering a sample of about 100 misclassified dev set examples, manually inspecting them to classify errors, counting their frequencies, and using these counts to prioritize fixing the most frequent categories first.

Explain error analysis priority decisions.

Case context:
You are building a classifier. You gather a sample of 100 dev set examples that your system misclassified. During manual error analysis, you count that 60% of these misclassified examples are dog images, whereas only 5% of them are cat images.

Question:
Based on the manual review of these 100 examples, which type of error should you prioritize fixing, and how does the error analysis justify this decision?

Sample answer:
You should prioritize fixing dog image misclassifications. The manual error analysis shows that dog images account for 60% of the error sample, while cat images account for only 5%. Prioritizing the dog category offers a significantly larger potential reduction in the overall error rate.

Key points:
- Identify dog images as the largest category of error (60% vs 5% for cats)
- Choose to prioritize dog images for fixing
- Ground the decision in the manual error analysis counts

Rubric:
The answer must identify dog images as the priority category and justify this choice by pointing out that they account for a much higher fraction (60%) of the errors in the manual sample of 100 misclassified examples compared to cats (5%).

Prioritizing fixes for an image classifier error

Question:
What is the purpose of manually reviewing about 100 dev set examples that the algorithm misclassifies?

Sample answer:
The purpose is to manually inspect where the algorithm is making mistakes, categorize these errors, and count their frequencies in order to prioritize which types of errors to work on fixing.

Key points:
- Inspect examples where the algorithm made an error
- Count the major categories of errors
- Prioritize what types of errors to work on fixing

Rubric:
The response must state that manually reviewing about 100 misclassified dev set examples allows the engineer to count major categories of errors to prioritize fixing them.

Learn Before

Related