Case context: A startup is building a model that detects leaks in industrial water pipes from sensor readings. The team has enough money to launch one product, but not enough to collect an unlimited amount of carefully labeled dev and test data. Leaders must decide whether to spend a small share or a large share of the budget on building these evaluation sets.

Question: Using the idea that investment in dev and test sets is a judgment call, what should the leadership weigh before deciding how much budget to devote to them?

Sample answer: They should judge how much a trustworthy evaluation process matters relative to the limited budget they have. They should think about the cost of mistakes in the real world, how hard it will be to gather and label representative data, and whether a smaller set can still reflect the kinds of cases the model will face after deployment.

Key points:

• Trade off spending against the need for dependable evaluation.
• Account for how serious errors would be in practice.
• Make sure the evaluation data resembles future operating conditions.

Rubric: The response should describe balancing limited resources with the need for accurate evaluation, while considering application risk and how well the dev/test sets represent future data.