Case context: A team has launched a text classifier and checked it on the dev set. The model produced 360 wrong predictions. One teammate proposes spending the next several days examining every mistaken example before deciding what to improve.

Question: If you are using an Eyeball dev set, what would you advise instead, and what is the reason?

Sample answer: I would start by examining a random set of about 50 errors rather than trying to inspect all 360 at once. A group of roughly 50 is usually enough to reveal the main patterns behind the mistakes. Looking through every error first would waste time, because the important failure types should become visible much sooner.

Key points:

• Suggest a random review of about 50 errors.
• Explain that this size is enough to uncover the main failure patterns.
• Note that reviewing all 360 immediately is not a good use of time.

Rubric: The answer should advise sampling about 50 errors and explain that this is enough to identify the main failure patterns while avoiding the time cost of checking all 360 examples.