Case context: A team is training an email classifier. It makes mistakes on 7% of the training examples, 8% on a holdout set drawn from the same source as the training data, and 17% on a development set collected from a different customer group.

Question: What is the most likely explanation for the model’s errors, and which concern is not supported by these results?

Sample answer: The results point to two problems. First, the model still has too much error even on data it saw during learning, which indicates avoidable bias. Second, the much larger error on the development set suggests a mismatch between the training-like data and the target distribution. What these numbers do not support is a large variance problem on the training-data distribution, because the training and same-source holdout errors are nearly the same.

Key points:

• Identify avoidable bias.
• Identify distribution mismatch.
• Do not infer large variance on the training-data distribution.

Rubric: A correct diagnosis will explicitly state that the model has avoidable bias and distribution mismatch, while also rejecting high variance on the training-data distribution because the training and same-source holdout errors are very close.