Case context: A team uses dev and test sets drawn from different distributions. A recent change improves performance on the dev set, but the team is unsure whether to make similar changes its next priority.

Question: What should the team diagnose as the source of its uncertainty, and why does it obstruct prioritization?

Sample answer: The team should diagnose the mismatch between the dev and test distributions as the source of uncertainty. Because a gain on the dev distribution may not improve test performance, the team cannot confidently conclude that the change is working for the test distribution. This makes it harder to decide whether related work deserves priority.

Key points:

• The dev and test distributions differ.
• A dev-set gain does not provide certainty of a test-set gain.
• The team has difficulty determining whether the change is working.
• The uncertainty weakens the basis for prioritization.

Rubric: The response should identify the dev/test distribution mismatch, explain uncertainty about transfer from dev to test performance, and connect that uncertainty to difficulty prioritizing work.