Question: Explain how manually reviewing about 100 misclassified dev set examples and counting error categories helps a machine learning team prioritize what types of errors to work on fixing.

Sample answer: Manually reviewing a sample of 100 misclassified dev set examples allows a team to classify the errors into distinct categories. By counting the fraction of errors in each category, the team can estimate the maximum impact of fixing each category. This quantitative feedback helps prioritize engineering work on the categories that cause the most errors.

Key points:

• Gather a sample of about 100 misclassified dev set examples
• Manually inspect and classify the errors into categories
• Count the frequency of each major error category
• Use the counts to prioritize engineering tasks on high-impact categories

Rubric: The response must describe gathering a sample of about 100 misclassified dev set examples, manually inspecting them to classify errors, counting their frequencies, and using these counts to prioritize fixing the most frequent categories first.