Question: Describe a situation in which the validation set and scoring metric point to one model as best, even though the team believes a different model would serve the product better. What is the clearest warning sign of this problem, and what are the three broad reasons it can happen?

Sample answer: The clearest warning sign is a mismatch between the model favored by the validation metric and the model the team would actually choose for the product. In other words, the numbers say one system is better, but practical judgment says another system is the better fit.

Three common reasons explain this:

1. The validation and test data do not reflect the real deployment setting, so the models are being judged on the wrong distribution.
2. The team has tuned repeatedly against the validation set, so performance on that set is no longer a reliable guide.
3. The metric captures only part of the real goal, so it rewards improvements that do not matter as much for the project.

Key points:

• Warning sign: the evaluation score prefers one model, but the team would deploy a different one.
• Cause 1: the data used for evaluation differs from the data the system will face in practice.
• Cause 2: repeated tuning has made the validation set less trustworthy.
• Cause 3: the metric is not aligned with the true product objective.

Rubric: A correct response must name the mismatch between metric-based ranking and the team’s preferred model as the warning sign, and must list all three causes: data-distribution mismatch, overfitting to the validation set, and a metric that does not fully match the project’s goal.