Question: How can close inspection of one development set lead a model team to overfit that set, and how does a separate untouched development set reveal that this has happened?

Sample answer: If engineers repeatedly study the same checked examples, they start learning patterns that are specific to those cases. They may tweak thresholds, add special rules, or adjust training choices so the model handles the inspected set better. Over time, that set becomes an easier target for improvement than data the team has not studied directly. A second development set that is never examined by hand provides a cleaner comparison. If scores rise much faster on the inspected set than on the untouched set, the team has evidence that it has adapted too closely to the inspected examples.

Key points:

• Repeated inspection gives the team detailed intuition about a small set of examples
• That intuition can drive choices that favor those examples specifically
• The untouched development set serves as a comparison point
• Faster gains on the inspected set than on the untouched set indicate overfitting
• The difference in progress is what reveals the problem

Rubric: The response must explain that repeated manual review encourages choices tailored to the inspected set, identify the untouched development set as a comparison baseline, and state that a much faster improvement on the inspected set signals overfitting.