Question: Discuss the recommended baseline number of errors to manually review in an Eyeball dev set and explain the condition under which reviewing significantly more errors is considered acceptable.

Sample answer: The recommended baseline is to manually review approximately 100 mistakes in an Eyeball dev set. This amount provides a very good sense of the major sources of errors the algorithm is making. However, some practitioners choose to analyze even more errors, sometimes up to 500. Doing so is acceptable and causes no harm as long as there is enough data available to support the expanded review.

Key points:

• Baseline of ~100 mistakes
• Provides a very good sense of major error sources
• Can review more errors (e.g., 500)
• Must have enough data to review more

Rubric: A strong answer will explicitly state the baseline of 100 mistakes, explain its purpose (finding major error sources), and correctly identify the condition for reviewing more errors (having enough data).