Google

If the initial dev/test set or metric missed the mark, change it quickly. One warning sign is that the dev set plus metric ranks one classifier higher while the team thinks another classifier is superior for the product. Possible causes include a dev/test set distribution that differs from the actual distribution of interest, overfitting to the dev set, or a metric that measures something other than what the project needs to optimize. After changing dev/test sets or metrics, make sure the team knows the new direction.

Changing Dev/Test Sets or the Metric When They No Longer Guide the Team

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A dev/test set can fail because its distribution is different from the actual distribution the system needs to perform well on. In that case, the dev/test sets should be updated to be more representative.

Dev/Test Set Distribution Not Representative of Actual Distribution

Repeatedly evaluating ideas on the dev set can cause an algorithm to gradually overfit to the dev set. If dev set performance is much better than test set performance when development is finished, that is a sign of dev-set overfitting. In that case, get a fresh dev set.

Dev Set Overfitting from Repeated Evaluation

A metric can fail by measuring something other than what the project needs to optimize. When this happens, the metric can no longer be trusted to pick the best algorithm, and it is time to change evaluation metrics.

Metric Optimizes the Wrong Project Objective

Which scenario is a warning sign that your dev/test set or evaluation metric needs to change?

True or False: Discovering that your initial dev/test set or metric missed the mark is a serious setback that cannot be easily corrected.

If your _____ is no longer measuring what is most important to you, Ng recommends changing it rather than continuing to optimize for it.

Which of the following is the key warning sign that your dev/test set or evaluation metric may need to be changed?

Ng considers discovering that a dev/test set or metric missed the mark to be a serious setback that requires restarting the evaluation process from scratch.

If your metric is no longer measuring what is most important to your project, you should change the _____.

Match each cause of a dev set/metric incorrectly ranking classifiers to the fix Ng recommends.

Order the steps a team should take upon discovering their dev/test set or metric is no longer guiding them correctly.

Your dev set contains formal customer emails but users primarily submit short social media posts. Which cause does this best illustrate?

After changing your dev/test sets or evaluation metric, updating the project files is sufficient — there is no need to explicitly inform the team of the new direction.

If you have overfit to the dev set, Ng recommends getting more _____ data.

Match each problem scenario to the cause category it represents in Ng's framework for incorrect classifier ranking.

Order the reasoning steps for deciding whether and how to change an evaluation metric that may no longer reflect project goals.

Question:
Explain how a development set and evaluation metric can incorrectly rank a worse classifier higher than a superior one. In your response, describe the main warning sign of this issue, and detail the three possible causes outlined by Andrew Ng.

Sample answer:
The main warning sign that the dev set and metric are not working correctly is when the dev set plus metric ranks one classifier higher, but the team thinks another classifier is superior for the product. The three main possible causes of this incorrect ranking are: 1) the actual distribution the team needs to do well on is different from the dev/test sets; 2) the team has overfit to the dev set; or 3) the evaluation metric is measuring something other than what the project needs to optimize.

Key points:
- The warning sign is when the dev set and metric rank one classifier higher than one the team knows is superior.
- Cause 1: The actual distribution differs from the dev/test set distribution.
- Cause 2: The model has overfit to the dev set.
- Cause 3: The metric measures something other than the project's optimization needs.

Rubric:
The response must identify the key warning sign (the dev set plus metric ranks a classifier higher but the team believes a different one is superior for the product) and list the three causes: distribution mismatch, overfitting to the dev set, and a metric that does not optimize what the project needs.

Analyze the warning signs and causes of a development set incorrectly ranking classifiers

Case context:
A development team is building a spam classifier. Their development set and accuracy metric rank Classifier A higher than Classifier B. However, the team manually reviews the results and realizes Classifier B is actually superior for the product because Classifier A lets too many highly offensive emails pass through. The team suspects their evaluation framework has missed the mark.

Question:
Given this scenario, identify the warning sign that occurred, name the most likely cause of the issue among the three outlined by Andrew Ng, and state what change the team must make and what action they must take immediately after.

Sample answer:
The warning sign is that the dev set plus metric ranks Classifier A higher, but the team thinks Classifier B is superior for the product. The most likely cause is that the evaluation metric is measuring something other than what the project needs to optimize (accuracy alone doesn't capture the product's need to block offensive emails). The team must quickly change the evaluation metric to align with their true objectives, and then make sure the entire team knows about the new direction.

Key points:
- Identify the discrepancy in classifier ranking as the key warning sign.
- Diagnose that the metric is no longer measuring what is most important to the project.
- Recommend changing the metric quickly.
- Ensure the team is informed of the new direction.

Rubric:
The response must identify the classifier ranking discrepancy as the warning sign, specify that the metric optimizes the wrong objective as the cause, recommend changing the metric, and state the necessity of informing the team about the new direction.

Diagnosing Classifier Ranking Mismatch in Spam Detection

Question:
According to Andrew Ng, if a machine learning team realizes that their dev set and metric are no longer pointing them in the right direction because they have overfit the dev set, what specific action should they take to resolve this issue?

Sample answer:
If the team has overfit the dev set, they should quickly change the dev set by getting more dev set data.

Key points:
- Identify overfitting to the dev set as the reason the dev set/metric is no longer guiding the team.
- Resolve the issue by getting more dev set data.

Rubric:
The answer must specify that the team should get more dev set data to resolve the overfitting issue.

Learn Before

Related