In a cat application, classification accuracy may rank classifier A above classifier B even though classifier A occasionally lets pornographic images through. In that situation, classifier A can be more accurate while still being unacceptable for the product.

Google

A metric can fail by measuring something other than what the project needs to optimize. When this happens, the metric can no longer be trusted to pick the best algorithm, and it is time to change evaluation metrics.

Metric Optimizes the Wrong Project Objective

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Pornographic Image Leakage as a Metric Failure Example

One way to change a failing evaluation metric is to heavily penalize letting through pornographic images.

Changing a Metric to Penalize Unacceptable Errors

When a metric is no longer trusted, choose a new metric and use it to explicitly define a new goal for the team, rather than proceeding for too long without a trusted metric and manually choosing among classifiers.

Learn Before

Related