Learn Before
Accuracy Is Not Always the Right Metric
In a pet-photo moderation app, one classifier can achieve higher overall accuracy than another and still be a poor choice if it sometimes approves violent images that should have been blocked. A model can score better on accuracy and still fail the product's real safety goal.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Accuracy Is Not Always the Right Metric
Using a Strong Penalty for a Critical Error Type
Set a New Evaluation Goal When the Current Metric Is Unreliable
What should a team do if its evaluation metric is rewarding the wrong outcome?
If an evaluation metric does not reflect the real project goal, it is still reliable for choosing the best model.
A score that measures the wrong target should not be used to ____ the best model.
Match each metric concept to its description.
What should a team do after noticing its metric points to the wrong goal?
What to do when an evaluation score favors the wrong goal
Diagnose a metric mismatch in a fraud detection model selection process.
What should a team do if its metric does not match the real objective?
What happens when an evaluation metric tracks the wrong goal?
A team should replace an evaluation metric that no longer reflects the project goal.
Learn After
In a family photo app example, why is classifier A still unacceptable even though its accuracy is higher?
True or False: A model with the highest overall accuracy is always the best choice for detecting a rare equipment defect.
A model can still be unacceptable if it misses the wrong kind of case
Match each item in the content-filter example to the statement that best describes it.
Order the reasoning that shows accuracy can be the wrong metric in a loan approval classifier example.
Why overall accuracy can miss the real goal in a content-filtering app
A radiology AI team must choose between two models after testing exposes a critical miss.
Why the higher-accuracy classifier is still rejected
What is the key lesson from this content moderation example about evaluation metrics?
True or False: Model 27 is the one that occasionally lets spam messages pass through.