Question: A fraud model is checked using transactions from large in-store purchases, but after deployment most flagged cases come from small mobile payments. What does that change imply about the validation set?

Sample answer: It means the validation and test data do not reflect the situations the model must handle after launch, so their evaluation results can be overly optimistic or otherwise misleading.

Key points:

• Validation/test data does not match the deployment setting
• Reported performance may not reflect real-world use

Rubric: Must state that the validation/test data does not match the target production distribution.