Question: A company builds an email classifier and evaluates it on a validation set and a test set that were both sampled from the same email stream. The model performs much better on validation than on test. Explain why this pattern points to overfitting the validation set, and why increasing the size of the validation set is the sensible remedy.

Sample answer: Because the validation and test sets come from the same distribution, the performance gap is not caused by the two sets representing different kinds of data. That rules out distribution mismatch as the main explanation. If the model looks strong on validation but weak on test, the most plausible conclusion is that repeated tuning has adapted the system too closely to the particular validation examples instead of to the underlying data source. In other words, the model has overfit the validation set. The practical fix is to collect more validation data, since a larger validation set gives a more reliable basis for model selection and makes it harder to chase quirks in a small sample.

Key points:

• Same-distribution validation and test data remove distribution mismatch as the explanation
• Better validation performance with worse test performance indicates overfitting to the validation set
• The diagnosis is specific: the model is tuned too closely to the validation examples
• More validation data is the appropriate cure because it improves the quality of model selection

Rubric: Full credit requires stating that shared distribution rules out data mismatch, identifying validation overfitting as the cause of the gap, and explaining why more validation data is the right fix.