Case context: A team working on a mature product recommendation system values a 0.01% accuracy improvement because it directly affects company profits. Its current dev set contains 10,000 examples.

Question: What should the team decide about dev set size, and why?

Sample answer: The team should consider using a dev set much larger than 10,000 examples. Because it wants to detect an improvement as small as 0.01%, the common range of 1,000 to 10,000 examples may not provide the needed sensitivity. The set should be enlarged enough to detect that meaningful change, without being made unnecessarily larger.

Key points:

• The target improvement is 0.01%.
• Important mature applications may value very small improvements.
• The dev set could be much larger than 10,000.
• Sizing should be driven by the change the team needs to detect.

Rubric: The response should recommend considering more than 10,000 examples, connect that recommendation to detecting a 0.01% improvement, and preserve the principle that the dev set only needs to be large enough for the meaningful change.