Case context: A product team is improving a fraud detection classifier and expects future model changes to raise validation accuracy by about 0.1%. Their development set contains 1,000 labeled examples.

Question: Assess whether a dev set with 1,000 examples is large enough to notice a 0.1% improvement, and state a concrete dev set size that would make such a difference more detectable.

Sample answer: A dev set of 1,000 examples is not large enough to dependably notice a 0.1% improvement. With only 1,000 cases, a 0.1% shift corresponds to a single example, which is too little to separate from ordinary variation. A dev set of about 10,000 examples would give a much better chance of detecting a 0.1% gain.

Key points:

• A 1,000-example dev set is still too small for a 0.1% improvement, even though it may be a reasonable general size.
• On a set of 1,000 examples, a 0.1% change is equivalent to just 1 example.
• Expanding the dev set to 10,000 examples is a practical target for noticing a 0.1% improvement.

Rubric: The student must explain that 1,000 examples is too small to reliably observe a 0.1% improvement and recommend increasing the dev set to 10,000 examples to make that change easier to detect.