Diagnose a scenario where actual user uploads differ from the dev/test set
Case context: A team discovers that their actual user uploads (the actual distribution they need to do well on) consist of images that are vastly different from the images they currently have in their dev and test sets.
Question: Based on this discovery, what is the underlying issue with their current evaluation setup, and what specific action should the team take to fix it?
Sample answer: The underlying issue is that their dev/test set distribution is not representative of the actual distribution they need to do well on. Because the actual distribution is different, the team should update their dev/test sets to be more representative of the actual user uploads.
Key points:
- The actual distribution is different from the dev/test sets.
- The current dev/test set distribution is not representative.
- The team must update the dev/test sets to be more representative.
Rubric: The student must identify that the dev/test set distribution is no longer representative of the actual distribution and recommend updating the dev/test sets.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Kitten Uploads Revealing Dev/Test Distribution Mismatch
When a team discovers the dev/test set distribution differs from actual deployment distribution, what is the recommended course of action?
True or False: A dev/test set that differs in distribution from actual deployment data still provides a reliable signal for model improvement.
When the dev/test set is not representative of the actual distribution, Ng recommends teams _____ the dev/test sets.
Match each term related to dev/test set distribution with its correct description.
Arrange the steps a team should follow after suspecting their dev/test set no longer matches the actual distribution.
Why does a dev/test set whose distribution mismatches the actual distribution fail to guide a development team effectively?
True or False: According to Machine Learning Yearning, the dev/test sets should reflect the actual distribution the system needs to perform well on.
According to Machine Learning Yearning p. 24: 'The actual distribution you need to do well on is _____ from the dev/test sets.'
Match each scenario to the correct conclusion about whether the dev/test set should be updated.
Arrange the logical reasoning steps that explain why a non-representative dev/test set must be updated.
Analyze the consequences of a mismatch between dev/test distribution and actual distribution
Diagnose a scenario where actual user uploads differ from the dev/test set
State the required action when a dev/test set distribution is not representative