Case context: A ranking team for an established movie-streaming platform cares about a 0.01% lift in accuracy because even that small change can noticeably affect subscription revenue. Its current development set has 10,000 examples.

Question: What should the team do about the development set, and what is the reason?

Sample answer: The team should strongly consider expanding the development set beyond 10,000 examples. When the goal is to notice an improvement as small as 0.01%, a set of only 10,000 examples may be too coarse to measure that change reliably. The right size is not chosen by a fixed rule; it should be large enough to make the desired improvement detectable, but not larger than necessary.

Key points:

• The team cares about a 0.01% improvement.
• Small but meaningful gains matter in mature systems.
• A 10,000-example set may not be sensitive enough here.
• The best size depends on the smallest change the team needs to observe.

Rubric: The response should recommend increasing the development set beyond 10,000 examples, explain that this is needed to detect a 0.01% improvement, and state that the set should be sized to match the meaningful change the team wants to measure.