Question: In a concise analytical response, explain why the size of a dev set should grow as the smallest accuracy gain worth noticing gets smaller. Use a support-ticket classifier and compare a 200-example set with a 20,000-example set.

Sample answer: A dev set is a measuring tool, so it must be large enough to separate real change from sampling noise. In a support-ticket classifier, 200 labeled examples are usually too few to trust a one-percentage-point accuracy difference. With about 20,000 examples, that same one-point gap is easier to distinguish. If the team only cares about a very tiny gain, such as 0.1 percentage point, the set has to be much larger still, because the uncertainty in measured accuracy shrinks slowly as the sample grows. The sensible goal is to make the dev set just large enough to guide model choice, not larger than needed.

Key points:

• The dev set should be large enough to detect the smallest useful accuracy gain.
• Small sets are too noisy for fine-grained comparisons.
• A 200-example set is weak evidence for a one-point difference.
• A 20,000-example set makes a one-point difference easier to detect.
• A 0.1 percentage-point target needs even more data.
• The set should be large enough for the decision, but not larger than necessary.

Rubric: A strong response explains that smaller target improvements require larger dev sets, uses the 200-example, 20,000-example, one-point, and 0.1 percentage-point scales correctly, and notes that the dev set only needs to be large enough for the intended comparison.