State the required action when a dev/test set distribution is not representative
Question: If a team realizes that the actual distribution their system needs to do well on is different from their dev/test sets, what should they do?
Sample answer: They should update their dev/test sets to be more representative of the actual distribution.
Key points:
- Acknowledge the mismatch between the dev/test sets and the actual distribution.
- Update the dev/test sets to be more representative.
Rubric: The answer must clearly state that the dev/test sets need to be updated to be more representative.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Kitten Uploads Revealing Dev/Test Distribution Mismatch
When a team discovers the dev/test set distribution differs from actual deployment distribution, what is the recommended course of action?
True or False: A dev/test set that differs in distribution from actual deployment data still provides a reliable signal for model improvement.
When the dev/test set is not representative of the actual distribution, Ng recommends teams _____ the dev/test sets.
Match each term related to dev/test set distribution with its correct description.
Arrange the steps a team should follow after suspecting their dev/test set no longer matches the actual distribution.
Why does a dev/test set whose distribution mismatches the actual distribution fail to guide a development team effectively?
True or False: According to Machine Learning Yearning, the dev/test sets should reflect the actual distribution the system needs to perform well on.
According to Machine Learning Yearning p. 24: 'The actual distribution you need to do well on is _____ from the dev/test sets.'
Match each scenario to the correct conclusion about whether the dev/test set should be updated.
Arrange the logical reasoning steps that explain why a non-representative dev/test set must be updated.
Analyze the consequences of a mismatch between dev/test distribution and actual distribution
Diagnose a scenario where actual user uploads differ from the dev/test set
State the required action when a dev/test set distribution is not representative