Dev and Test Set Design for Mobile Image Applications
Case context: A team is building a cat detector app. They have collected 100,000 website images for training. However, the app will run on mobile devices, and the team expects the future data from mobile phone images to differ in nature from the website images. The team decides to take 30% of the website images to serve as their dev and test sets.
Question: According to Machine Learning Yearning, diagnose the problem with the team's split strategy and explain how they should choose their dev and test sets instead.
Sample answer: The problem is that the dev and test sets are taken from the website images distribution, which differs from the mobile phone images distribution expected in the future. The team should not assume the training distribution is the same as the test distribution, nor should they simply split 30% of the available training data. Instead, they should choose dev and test sets that reflect the mobile phone images they expect to receive in the future and want the system to perform well on.
Key points:
- A 30% random split of website images does not represent the mobile phone image distribution.
- Dev and test sets should be chosen to reflect future data (mobile phone images) rather than training data (website images).
- The team must design the evaluation sets around the distribution they want to perform well on.
Rubric: The answer must identify that a 30% split of website images is problematic because mobile phone images differ in distribution, and state that dev/test sets must reflect the future mobile phone image distribution the system is targeted to perform well on.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Adding More Training Data Does Not Always Help
Special Challenges from Different Training and Dev/Test Distributions
Risk of Merging Training Data Sources Depends on Algorithm Flexibility
Shared Label Mapping Across Data Sources
Training and Dev/Test Sets from Different Distributions
Inconsistent Auxiliary Data Source
Approximating Future Dev/Test Data Before Launch
Updating Dev/Test Sets with Actual User Data After Launch
Risk of Starting with Website Images When Future-Like Data Is Unavailable
Development Investment for Dev and Test Sets Requires Judgment
According to Machine Learning Yearning, what is the primary criterion for choosing dev and test sets?
True or False: When building a dev/test set, it is safe to assume the training distribution is the same as the test distribution.
Dev and test sets should contain examples that reflect what you ultimately want to perform well on, rather than only the _____ you happen to have for training.
Why is using a simple 30% random split of available data as your test set problematic when future data differs from training data?
According to ML Yearning, it is generally safe to assume your training data distribution is the same as your test data distribution.
Dev and test sets should be chosen to reflect data you expect to get in the _____ and want to do well on.
Match each dev/test set concept from ML Yearning to its correct description.
Order the steps for correctly choosing dev and test sets according to ML Yearning's guidance.
According to ML Yearning, what should the examples in your dev and test sets primarily reflect?
According to ML Yearning, dev and test sets must always come from the same distribution as the training data.
ML Yearning warns that the test set should not simply be _____ of the available data when future data differs from the training set.
Match each data scenario to the correct dev/test set strategy decision according to ML Yearning.
Order the reasoning steps for deciding whether a proposed dev/test set is well-chosen, per ML Yearning.
Why Standard Data Splits Fail With Different Future Distributions
Dev and Test Set Design for Mobile Image Applications
The Core Criterion for Dev and Test Set Selection