Analyze the consequences of a mismatch between dev/test distribution and actual distribution
Question: Discuss why a machine learning system's dev/test set might fail to guide the development team effectively if its distribution differs from the actual distribution, and state the required action.
Sample answer: If the dev/test set distribution is not representative of the actual distribution the system needs to perform well on, the evaluation metrics will not reflect reality. This mismatch misguides the team because optimizing for the dev/test set will not improve real-world performance. When this occurs, the team must update the dev/test sets so they are more representative of the actual distribution.
Key points:
- The actual distribution you need to do well on can be different from the dev/test sets.
- This makes the dev/test set distribution not representative of the actual distribution.
- The required action is to update the dev/test sets to be more representative.
Rubric: The response should explain that a non-representative dev/test set gives misleading performance signals because it differs from the actual distribution, and state that the sets must be updated to be more representative.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Kitten Uploads Revealing Dev/Test Distribution Mismatch
When a team discovers the dev/test set distribution differs from actual deployment distribution, what is the recommended course of action?
True or False: A dev/test set that differs in distribution from actual deployment data still provides a reliable signal for model improvement.
When the dev/test set is not representative of the actual distribution, Ng recommends teams _____ the dev/test sets.
Match each term related to dev/test set distribution with its correct description.
Arrange the steps a team should follow after suspecting their dev/test set no longer matches the actual distribution.
Why does a dev/test set whose distribution mismatches the actual distribution fail to guide a development team effectively?
True or False: According to Machine Learning Yearning, the dev/test sets should reflect the actual distribution the system needs to perform well on.
According to Machine Learning Yearning p. 24: 'The actual distribution you need to do well on is _____ from the dev/test sets.'
Match each scenario to the correct conclusion about whether the dev/test set should be updated.
Arrange the logical reasoning steps that explain why a non-representative dev/test set must be updated.
Analyze the consequences of a mismatch between dev/test distribution and actual distribution
Diagnose a scenario where actual user uploads differ from the dev/test set
State the required action when a dev/test set distribution is not representative