Case context: A team is improving a customer-support ticket classifier. Their validation set is large, and the model is making many mistakes. The lead scientist wants to inspect 400 misclassified examples to get a very detailed picture of the failure patterns, while another engineer says that 120 examples are enough.

Question: Using the idea of a manual error-review set, assess the lead scientist's plan. Is it reasonable, and why?

Sample answer: Yes, the plan is reasonable. Reviewing about 120 mistakes is often enough to identify the major error sources, but examining 400 mistakes is also acceptable. The main requirement is that the team has enough data to support the extra analysis, and in this case the dataset is large enough.

Key points:

• Around 120 errors is enough to see the main patterns
• Reviewing 400 errors is acceptable
• The team has enough data for the larger review

Rubric: The answer should explain that 120 examples can reveal the main error sources, while also stating that 400 examples is fine when the dataset is large enough.