Case context: A startup builds a detector for the phrase "start recording." Most training examples are clean studio clips collected from volunteers. However, the product will be used in cars, where voices are often mixed with road noise, music, and changing microphone quality. The model looks strong on the current evaluation set but fails on real in-car recordings.

Question: What problem should the team identify, and how should it change its dev and test sets?

Sample answer: The team should identify that its evaluation data comes from a cleaner distribution than the one the product must handle. It should rebuild the dev and test sets from in-car voice recordings because those match the future operating conditions and the distribution that matters for product performance. The clean studio clips can still be valuable as extra training data.

Key points:

• Clean studio clips are not the same as in-car recordings.
• The evaluation set does not match the target use case.
• Dev and test data should come from in-car conditions.
• Studio clips may still help during training.

Rubric: The answer should state that the evaluation set is drawn from the wrong distribution, choose in-car recordings for dev and test, explain the choice in terms of the intended deployment setting, and not say that the clean studio data must be removed from training.