Learn Before
How to Repair a Metric That Misses Critical Errors
Question: If an evaluation metric still rates a model too highly even when it allows an unacceptable mistake, such as approving a shipment with a dangerous safety defect, what adjustment should you make?
Sample answer: Change the metric so that the unacceptable mistake receives a much larger penalty, either by adding a strong penalty term or by increasing its weight in the score.
Key points:
- Update the metric rather than leaving it unchanged.
- Assign a heavy penalty to the specific unacceptable error.
Rubric: The response should say that the metric must be modified to strongly penalize the particular unacceptable error, such as a dangerous safety defect.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
How should a metric change when it fails to penalize a serious mistake?
True or False: Giving a larger penalty to especially serious mistakes can make an evaluation metric better reflect the actual project goal.
To make a metric reflect an unacceptable mistake, you can redesign it to heavily _____ that mistake.
How should an evaluation metric be changed when it misses the real cost of certain mistakes?
A Metric Can Be Adjusted to Reflect the Real Goal
Strengthening a Metric Around a Critical Mistake
Match each term to its definition when revising an evaluation score that rewards the wrong behavior.
Order the steps for correcting a metric that rewards the wrong classifier behavior.
Which example illustrates changing a metric to strongly penalize an unacceptable error?
If a metric misses the real objective, weighting every mistake the same usually fixes it.
Changing a metric to penalize unacceptable mistakes is a way to fix a metric that rewards the wrong project _____.
Match each case to its role in changing a metric so it punishes severe mistakes.
Order the steps for revising a metric to reflect a serious error.
Designing a cost-sensitive metric for high-risk classification errors
Choosing a Metric for a Moderation System with Severe Mistakes
How to Repair a Metric That Misses Critical Errors