Question: In machine learning systems, an evaluation metric might classify all classification errors equally, even when some errors (such as letting through pornographic images) are unacceptable to users. Explain how and why a practitioner would modify the evaluation metric to address this discrepancy, and discuss the implications of assigning a heavy penalty weight to these specific unacceptable errors.

Sample answer: To correct a failing evaluation metric that treats all errors equally, a practitioner can modify the metric's formula to heavily penalize unacceptable errors, such as letting through pornographic images. By assigning a large weight (penalty) to these specific false positives or false negatives, the metric is aligned with the actual project objective. This ensures that models which commit highly objectionable errors receive a much poorer score during evaluation, steering the model selection process toward safer and more acceptable performance.

Key points:

• Modifying the evaluation metric by adding a heavy penalty weight to unacceptable errors.
• Aligning the metric with the actual project objective when standard classification error treating is insufficient.
• Using the specific example of heavily penalizing pornographic images to protect user experience.
• Influencing model selection to prefer models that avoid high-cost, unacceptable mistakes.

Rubric: The response must explain how the evaluation metric is modified (by adding a heavy penalty/weight to specific errors), why this modification is necessary (to align the metric with the project's real-world objective and prevent unacceptable errors like pornographic images), and how this affects model selection (by penalizing and filtering out models that make these unacceptable errors).