Case context: A team has been using a 100-example Eyeball dev set for weeks, manually reviewing errors after every model iteration. They notice their model now performs very well on this Eyeball dev set, but its performance on the separate, non-inspected Blackbox dev set has stayed flat and is noticeably worse.

Question: What should the team conclude about their Eyeball dev set, and what action should they take to address the problem?

Sample answer: The team should conclude that their Eyeball dev set has been overfit, since performance on it is much better than on the Blackbox dev set. This happened because repeated manual error analysis and tuning decisions were made specifically to address errors seen in that small set, causing the process to fit its idiosyncrasies rather than generalize. To remedy this, the team should consider acquiring more data for the Eyeball dev set, either by moving more examples from the Blackbox dev set or by labeling new data, so that future error analysis is based on a larger, more representative sample.

Key points:

• The performance gap indicates the Eyeball dev set is overfit
• Overfitting resulted from repeated manual tuning on a small, fixed set
• The Blackbox dev set reveals the true, more objective performance
• The recommended remedy is acquiring more data for the Eyeball dev set

Rubric: Full credit requires correctly diagnosing overfitting based on the performance gap and recommending acquiring more data as the remedy.