Analyze the warning signs and causes of a development set incorrectly ranking classifiers
Question: Explain how a development set and evaluation metric can incorrectly rank a worse classifier higher than a superior one. In your response, describe the main warning sign of this issue, and detail the three possible causes outlined by Andrew Ng.
Sample answer: The main warning sign that the dev set and metric are not working correctly is when the dev set plus metric ranks one classifier higher, but the team thinks another classifier is superior for the product. The three main possible causes of this incorrect ranking are: 1) the actual distribution the team needs to do well on is different from the dev/test sets; 2) the team has overfit to the dev set; or 3) the evaluation metric is measuring something other than what the project needs to optimize.
Key points:
- The warning sign is when the dev set and metric rank one classifier higher than one the team knows is superior.
- Cause 1: The actual distribution differs from the dev/test set distribution.
- Cause 2: The model has overfit to the dev set.
- Cause 3: The metric measures something other than the project's optimization needs.
Rubric: The response must identify the key warning sign (the dev set plus metric ranks a classifier higher but the team believes a different one is superior for the product) and list the three causes: distribution mismatch, overfitting to the dev set, and a metric that does not optimize what the project needs.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Dev/Test Set Distribution Not Representative of Actual Distribution
Dev Set Overfitting from Repeated Evaluation
Metric Optimizes the Wrong Project Objective
Which scenario is a warning sign that your dev/test set or evaluation metric needs to change?
True or False: Discovering that your initial dev/test set or metric missed the mark is a serious setback that cannot be easily corrected.
If your _____ is no longer measuring what is most important to you, Ng recommends changing it rather than continuing to optimize for it.
Which of the following is the key warning sign that your dev/test set or evaluation metric may need to be changed?
Ng considers discovering that a dev/test set or metric missed the mark to be a serious setback that requires restarting the evaluation process from scratch.
If your metric is no longer measuring what is most important to your project, you should change the _____.
Match each cause of a dev set/metric incorrectly ranking classifiers to the fix Ng recommends.
Order the steps a team should take upon discovering their dev/test set or metric is no longer guiding them correctly.
Your dev set contains formal customer emails but users primarily submit short social media posts. Which cause does this best illustrate?
After changing your dev/test sets or evaluation metric, updating the project files is sufficient — there is no need to explicitly inform the team of the new direction.
If you have overfit to the dev set, Ng recommends getting more _____ data.
Match each problem scenario to the cause category it represents in Ng's framework for incorrect classifier ranking.
Order the reasoning steps for deciding whether and how to change an evaluation metric that may no longer reflect project goals.
Analyze the warning signs and causes of a development set incorrectly ranking classifiers
Diagnosing Classifier Ranking Mismatch in Spam Detection
Response to Overfitting the Development Set