Case context: Your team is building a system that turns recorded meetings into written summaries. You divide the development data into two parts: a Review dev set that the team inspects by hand to understand mistakes, and a Shield dev set that is used only for evaluation. After two weeks of tuning preprocessing choices and model settings based on the manual review, the error rate on the Review dev set falls from 12% to 4%, while the Shield dev set error rate falls only from 12% to 11%.

Question: What does this pattern suggest about the Review dev set, and what next step should the team consider?

Sample answer: This pattern suggests that the Review dev set has been overfit through repeated manual analysis. The much larger improvement on the Review set than on the Shield set shows that the team has started optimizing too closely to the inspected data. A reasonable next step is to collect more data for the Review dev set or otherwise refresh it so it better represents the problem being evaluated.

Key points:

• Identify that the Review dev set has likely been overfit.
• Note that the main clue is the large gap between improvement on the Review set and improvement on the Shield set.
• Recommend getting more data for the Review dev set as a possible fix.

Rubric: The response must correctly diagnose overfitting to the Review dev set using the contrast between the two error-rate changes, and it must suggest acquiring more Review dev set data or an equivalent remedy to reduce the mismatch.