Designing a Safe Image Filter Metric for a Public Platform
Case context: You are developing a child-friendly content filter for an image sharing application. The current evaluation metric treats a standard misclassification (e.g., misclassifying a cat as a dog) and an unacceptable error (e.g., letting through a pornographic image) with equal weight. Consequently, the team is selecting models that occasionally let through inappropriate content because their overall accuracy is high.
Question: Based on the concept of changing a metric to penalize unacceptable errors, what decision should you make regarding the evaluation metric to resolve this issue?
Sample answer: You should modify the evaluation metric to heavily penalize the unacceptable error of letting through pornographic images. Instead of treating all misclassifications equally, you should introduce a high weight or penalty factor specifically for instances where inappropriate content is classified as safe, ensuring that any model committing this error is penalized severely and not selected for deployment.
Key points:
- Identify that the evaluation metric fails to reflect the severity of different error types.
- Propose changing the metric by introducing a heavy penalty specifically for pornographic images.
- Explain that the modified metric will prevent the selection of models that permit unacceptable errors.
Rubric: The answer must identify the need to change the evaluation metric, specify that a heavy penalty must be applied to the unacceptable error (letting through pornographic images), and explain how this penalty guides the model selection process away from models that leak inappropriate content.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Related
When a metric optimizes the wrong project objective, what is one recommended way to fix it?
True or False: Heavily penalizing unacceptable errors in a metric is a valid way to realign the metric with the true project objective.
To fix a metric that optimizes the wrong objective, you can change the metric to heavily _____ the specific unacceptable error type.
When an evaluation metric optimizes the wrong project objective, what does Ng recommend to fix it?
Heavily penalizing certain error types in an evaluation metric can correct a metric that optimizes the wrong objective.
According to Ng, one way to change a failing evaluation metric is to heavily _____ letting through pornographic images.
Match each term to its definition in the context of fixing a metric that optimizes the wrong objective.
Order the steps for fixing an evaluation metric that optimizes the wrong objective by penalizing unacceptable errors.
What specific example does Ng use in Machine Learning Yearning (p. 25) to illustrate modifying a metric with a heavy penalty?
When a metric fails to optimize the correct objective, treating all error types with equal weight is an adequate solution.
Modifying a metric to penalize unacceptable errors is a technique for correcting a metric that optimizes the wrong project _____.
Match each scenario to the role it plays in Ng's strategy of penalizing unacceptable errors to fix a metric.
Order the reasoning steps a practitioner follows when deciding to add a heavy penalty for a specific error type in an evaluation metric.
Analyzing the Impact of Custom Penalty Weights in Evaluation Metrics for User Safety
Designing a Safe Image Filter Metric for a Public Platform
Mechanism for Correcting an Evaluation Metric that Fails to Penalize Unacceptable Errors