Question: In a concise analytical response, explain why teams targeting smaller accuracy improvements may require larger dev sets. Use the 0.1% and 0.01% examples from the source.

Sample answer: Dev set size should reflect the smallest accuracy change the team considers meaningful. A 100-example dev set cannot detect a difference such as 90.0% versus 90.1%, while 10,000 examples gives a good chance of detecting a 0.1% improvement. Mature and important applications may value improvements as small as 0.01%, so their dev sets could be much larger than 10,000. However, a dev set need not be much larger than what is required to detect the meaningful change.

Key points:

• Dev set size depends on the meaningful accuracy change.
• One hundred examples cannot detect 90.0% versus 90.1%.
• Ten thousand examples gives a good chance of detecting a 0.1% improvement.
• Detecting a 0.01% improvement may require much more than 10,000 examples.
• The dev set need not be much larger than necessary.

Rubric: A strong response connects smaller target improvements to larger dev sets, accurately uses the 100-example, 10,000-example, 0.1%, and 0.01% evidence, and notes that unnecessary additional size is not required.