Question: A team has abundant labeled data from an auxiliary source, but only a limited amount of data that matches the distribution of its dev and test sets. Why should some of the target-distribution examples still be used for training instead of reserving all of them for evaluation? Explain how this choice affects learning and how it helps the team decide whether more target-like data would be useful.

Sample answer: Using some target-distribution examples for training helps the model learn patterns that matter in the environment where it will actually be judged. If training relies only on auxiliary data, the model may fit the wrong distribution and perform poorly on the target domain.

It is also helpful to keep target-distribution examples in both the training set and a training-dev set. The training set shows whether the model can learn from the target domain, while the training-dev set gives a cleaner estimate of performance on that same domain. If performance is much better on the target examples used for training than on the target training-dev set, that suggests the model is not yet capturing the target distribution well and that collecting more target-like data may improve results.

Key points:

• Some target-distribution data should be used for training so the model learns the right domain.
• A target-distribution training-dev split lets the team measure performance on that domain during development.
• A gap between target training performance and target training-dev performance can indicate a need for more target-like data.

Rubric: The response should explain that target-distribution examples help the model learn domain-specific patterns. It should also describe the value of evaluating target-distribution performance during training and comparing target training versus target training-dev results to judge whether more target-like data would likely help.