Question: Using the cat application example, explain why classification accuracy failed to capture the true product objective, and describe what this reveals about choosing metrics for a machine learning product.

Sample answer: Classification accuracy only measures the overall proportion of correct predictions, so it treats all errors as equally important. In the cat app, classifier A had higher accuracy but occasionally let pornographic images through, an error type with severe consequences for user trust that accuracy does not weigh differently from any other misclassification. This shows that a metric must reflect the true costs and priorities of the product, not just an aggregate correctness score.

Key points:

• Classification accuracy aggregates all errors equally
• Classifier A had higher accuracy but let pornographic images through
• This error type has disproportionately severe consequences
• The metric did not reflect the true product objective
• Metrics must be chosen to capture real-world costs, not just overall correctness

Rubric: Full credit requires identifying that accuracy treats all errors equally, explaining that pornographic leakage is a high-cost error type, and connecting this to the need for metrics aligned with true product objectives.