Case context: A payments company is comparing several fraud detection models. It selects the model with the highest overall accuracy on a validation set. In practice, the chosen model misses many fraudulent transactions because fraud is rare and the metric rewards mostly correct predictions on legitimate transactions.

Question: Explain why the current selection process is failing and state what the team should change.

Sample answer: The process is failing because overall accuracy does not reflect the project's real goal, which is to catch fraud effectively. A model can score well on accuracy while still missing too many bad transactions. The team should replace the current metric with one that matches the business objective, such as a metric that emphasizes fraud detection or the cost of missed fraud.

Key points:

• The metric measures the wrong thing for the business goal.
• High accuracy can hide poor fraud detection when the class is imbalanced.
• The team should use a metric aligned with the true objective, especially the cost of missed fraud.

Rubric: The student must identify that the metric is misaligned with the project objective and recommend changing to a metric that better reflects fraud detection performance.