Question: Explain why repeatedly checking ideas against a development set can make that set less reliable. Then describe how to tell this has happened at the end of development and what should be done next.

Sample answer: If a team keeps using the same development set to compare many ideas, the design process can gradually adapt to that set's quirks, which makes the development set an overly optimistic estimate of real performance. At the end of development, you diagnose this by comparing development-set results with test-set results; if the development score is much better, the development set has been overused. The remedy is to replace it with a newly collected development set and evaluate ideas against the new one.

Key points:

• Repeated evaluation on the same development set can cause overfitting to that set.
• The issue is diagnosed by comparing development and test performance after development is complete.
• A large gap with better development performance is a warning sign.
• The fix is to obtain a fresh development set.

Rubric: The student should state the cause, explain the comparison used for diagnosis, and identify replacing the development set as the solution.