Google

A metric can fail by measuring something other than what the project needs to optimize. When this happens, the metric can no longer be trusted to pick the best algorithm, and it is time to change evaluation metrics.

Metric Optimizes the Wrong Project Objective

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

In a cat application, classification accuracy may rank classifier A above classifier B even though classifier A occasionally lets pornographic images through. In that situation, classifier A can be more accurate while still being unacceptable for the product.

Pornographic Image Leakage as a Metric Failure Example

One way to change a failing evaluation metric is to heavily penalize letting through pornographic images.

Changing a Metric to Penalize Unacceptable Errors

When a metric is no longer trusted, choose a new metric and use it to explicitly define a new goal for the team, rather than proceeding for too long without a trusted metric and manually choosing among classifiers.

Choosing a New Trusted Metric Instead of Manual Classifier Selection

What action should a team take when their evaluation metric no longer aligns with the project's actual objectives?

Can a metric that measures the wrong objective still be trusted to select the best machine learning algorithm?

If a metric optimizes the wrong objective, you can no longer trust it to _____ the best algorithm.

Match metric issues and actions to their correct descriptions.

What is the correct sequence of decisions when a team discovers a metric alignment issue?

Question:
Explain what happens when a machine learning evaluation metric measures something other than what a project needs to optimize, and describe the action that must be taken to resolve this issue.

Sample answer:
When a metric measures something other than the project's actual goals, it can no longer be trusted to select the best algorithm for the project. Because the metric is guiding the team toward the wrong objective, the team must change evaluation metrics to one that accurately reflects the project's needs.

Key points:
- The metric measures something other than the project's optimization needs.
- The team can no longer trust the metric to pick the best algorithm.
- The correct resolution is to change the evaluation metrics.

Rubric:
The response should clearly identify that (1) the metric ceases to align with project needs, (2) the metric cannot be trusted to pick the best algorithm, and (3) the correct action is to change the evaluation metric.

Analyze the consequences and resolution when a machine learning metric optimizes the wrong objective.

Case context:
A machine learning team has been using a classification metric to select their final models. However, they discover that the selected models frequently perform poorly in the actual product environment because the metric is measuring something other than what the project needs to optimize.

Question:
Based on the source concepts, diagnose why the team's current model selection process is failing and recommend the appropriate action.

Sample answer:
The process is failing because the evaluation metric does not align with the project's actual optimization needs, meaning it can no longer be trusted to pick the best algorithm. The team must change their evaluation metric to one that accurately reflects their product objectives.

Key points:
- The metric is measuring something other than the project's optimization needs.
- The metric can no longer be trusted to pick the best algorithm.
- The team must change the evaluation metric to guide the project correctly.

Rubric:
The student must diagnose that the metric cannot be trusted to select the best algorithm due to misaligned objectives, and recommend changing the evaluation metric.

Diagnose a metric alignment issue in a classifier selection process.

Question:
When a metric is measuring something other than what the project needs to optimize, what happens to the trust in the metric and what should the team do?

Sample answer:
The metric can no longer be trusted to pick the best algorithm. Consequently, the team must change the evaluation metric to align with the project's needs.

Key points:
- The metric can no longer be trusted to pick the best algorithm.
- It is time to change evaluation metrics.

Rubric:
The answer should state that the metric cannot be trusted to select the best algorithm and that the team needs to change evaluation metrics.

Learn Before

Related