Question: In a data-mismatch setting, why can performance improvement become unclear if you cannot obtain training examples that resemble the dev set?

Sample answer: When the training distribution is different from the dev distribution, the model may fail because it has not learned from examples like the ones it will be judged on. The most direct fix is to add or create training data that is closer to the dev set. If that is not possible, then the main cause of the error is still untouched. Since this kind of adaptation does not come with a guaranteed remedy, you may reach a point where no obvious data or modeling change is left to close the gap.

Key points:

• A mismatch is best addressed by making training examples more like the dev examples.
• Without such data, the model cannot learn the target distribution well.
• Improving performance in this situation is not guaranteed.

Rubric: The essay should explain that the absence of matching training data prevents the model from learning the target distribution and leaves no certain direct fix.