Question: A team is building a loan-fraud detector. They manually inspect and fix mislabeled examples in the validation set, but they leave the test set labels untouched. Explain the consequences for model selection and final evaluation.

Sample answer: Cleaning only the validation set changes the standard used to judge models during development, while the test set still uses the old labels. As a result, the validation set and test set no longer measure the same thing. The team will tune the model to do well on the cleaned validation set, but the final score will come from the untouched test set. That mismatch can make a model look strong during development even though it does not achieve the same result on final evaluation.

Key points:

• Cleaning only the validation set makes the validation and test sets inconsistent.
• The team optimizes model choices using the cleaned validation set.
• Final performance is judged with the unchanged test set.
• Because the criteria differ, validation improvement may not carry over to test performance.

Rubric: The answer must state that fixing labels only in the validation set creates a mismatch between validation and test evaluation. It must also explain that model selection is then guided by one criterion while final test scoring uses another.