Question: In the context of error analysis, detail how a classifier's error rate impacts the required size of the Eyeball dev set. Provide a mathematical example demonstrating how to calculate the required dev set size to obtain an adequate number of errors for manual inspection.

Sample answer: The classifier's error rate and the required Eyeball dev set size have an inverse relationship. When the error rate is lower, a larger Eyeball dev set is necessary to capture enough misclassified examples for meaningful manual analysis (which is typically around 100 errors). For instance, if a classifier has a 5% error rate, the overall dev set needs to contain roughly 2,000 examples so that 5% of them yield about 100 misclassified examples (0.05 * 2,000 = 100) to inspect.

Key points:

• Inverse relationship between error rate and Eyeball dev set size
• Goal is to accumulate a sufficient number of errors (around 100) for manual analysis
• Example calculation showing error rate (0.05) multiplied by set size (2000) equals desired number of errors (100)

Rubric: The answer must clearly state the inverse relationship, explain the goal of obtaining enough misclassified examples (approximately 100), and accurately demonstrate the mathematical calculation using the 5% error rate example.