Dev/Test Set Distribution Not Representative of Actual Distribution
A dev/test set can fail because its distribution is different from the actual distribution the system needs to perform well on. In that case, the dev/test sets should be updated to be more representative.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Related
Dev/Test Set Distribution Not Representative of Actual Distribution
Dev Set Overfitting from Repeated Evaluation
Metric Optimizes the Wrong Project Objective
Which scenario is a warning sign that your dev/test set or evaluation metric needs to change?
True or False: Discovering that your initial dev/test set or metric missed the mark is a serious setback that cannot be easily corrected.
If your _____ is no longer measuring what is most important to you, Ng recommends changing it rather than continuing to optimize for it.
Which of the following is the key warning sign that your dev/test set or evaluation metric may need to be changed?
Ng considers discovering that a dev/test set or metric missed the mark to be a serious setback that requires restarting the evaluation process from scratch.
If your metric is no longer measuring what is most important to your project, you should change the _____.
Match each cause of a dev set/metric incorrectly ranking classifiers to the fix Ng recommends.
Order the steps a team should take upon discovering their dev/test set or metric is no longer guiding them correctly.
Your dev set contains formal customer emails but users primarily submit short social media posts. Which cause does this best illustrate?
After changing your dev/test sets or evaluation metric, updating the project files is sufficient — there is no need to explicitly inform the team of the new direction.
If you have overfit to the dev set, Ng recommends getting more _____ data.
Match each problem scenario to the cause category it represents in Ng's framework for incorrect classifier ranking.
Order the reasoning steps for deciding whether and how to change an evaluation metric that may no longer reflect project goals.
Analyze the warning signs and causes of a development set incorrectly ranking classifiers
Diagnosing Classifier Ranking Mismatch in Spam Detection
Response to Overfitting the Development Set
Learn After
Kitten Uploads Revealing Dev/Test Distribution Mismatch
When a team discovers the dev/test set distribution differs from actual deployment distribution, what is the recommended course of action?
True or False: A dev/test set that differs in distribution from actual deployment data still provides a reliable signal for model improvement.
When the dev/test set is not representative of the actual distribution, Ng recommends teams _____ the dev/test sets.
Match each term related to dev/test set distribution with its correct description.
Arrange the steps a team should follow after suspecting their dev/test set no longer matches the actual distribution.
Why does a dev/test set whose distribution mismatches the actual distribution fail to guide a development team effectively?
True or False: According to Machine Learning Yearning, the dev/test sets should reflect the actual distribution the system needs to perform well on.
According to Machine Learning Yearning p. 24: 'The actual distribution you need to do well on is _____ from the dev/test sets.'
Match each scenario to the correct conclusion about whether the dev/test set should be updated.
Arrange the logical reasoning steps that explain why a non-representative dev/test set must be updated.
Analyze the consequences of a mismatch between dev/test distribution and actual distribution
Diagnose a scenario where actual user uploads differ from the dev/test set
State the required action when a dev/test set distribution is not representative