Question: What is the purpose of manually reviewing about 100 dev set examples that the algorithm misclassifies?

Sample answer: The purpose is to manually inspect where the algorithm is making mistakes, categorize these errors, and count their frequencies in order to prioritize which types of errors to work on fixing.

Key points:

• Inspect examples where the algorithm made an error
• Count the major categories of errors
• Prioritize what types of errors to work on fixing

Rubric: The response must state that manually reviewing about 100 misclassified dev set examples allows the engineer to count major categories of errors to prioritize fixing them.