Mechanism for Correcting an Evaluation Metric that Fails to Penalize Unacceptable Errors
Question: According to the course, what specific adjustment can you make to a failing evaluation metric if it currently fails to prevent highly undesirable mistakes like letting through pornographic images?
Sample answer: You can modify the evaluation metric by introducing a heavy penalty (or a large weight multiplier) for those specific unacceptable errors, such as letting through pornographic images, so that the metric correctly penalizes models that commit these mistakes.
Key points:
- Modify the failing evaluation metric.
- Heavily penalize the unacceptable errors (e.g., pornographic images).
Rubric: The response should state that the metric must be changed to heavily penalize the specific unacceptable error (such as pornographic images).
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Related
When a metric optimizes the wrong project objective, what is one recommended way to fix it?
True or False: Heavily penalizing unacceptable errors in a metric is a valid way to realign the metric with the true project objective.
To fix a metric that optimizes the wrong objective, you can change the metric to heavily _____ the specific unacceptable error type.
When an evaluation metric optimizes the wrong project objective, what does Ng recommend to fix it?
Heavily penalizing certain error types in an evaluation metric can correct a metric that optimizes the wrong objective.
According to Ng, one way to change a failing evaluation metric is to heavily _____ letting through pornographic images.
Match each term to its definition in the context of fixing a metric that optimizes the wrong objective.
Order the steps for fixing an evaluation metric that optimizes the wrong objective by penalizing unacceptable errors.
What specific example does Ng use in Machine Learning Yearning (p. 25) to illustrate modifying a metric with a heavy penalty?
When a metric fails to optimize the correct objective, treating all error types with equal weight is an adequate solution.
Modifying a metric to penalize unacceptable errors is a technique for correcting a metric that optimizes the wrong project _____.
Match each scenario to the role it plays in Ng's strategy of penalizing unacceptable errors to fix a metric.
Order the reasoning steps a practitioner follows when deciding to add a heavy penalty for a specific error type in an evaluation metric.
Analyzing the Impact of Custom Penalty Weights in Evaluation Metrics for User Safety
Designing a Safe Image Filter Metric for a Public Platform
Mechanism for Correcting an Evaluation Metric that Fails to Penalize Unacceptable Errors