Do Not Randomly Mix Sources When Building Validation and Test Sets
If your data comes from sources with different distributions, a random split of all examples can make validation and test sets unrepresentative of the data you care about later. Those sets should match the distribution expected in deployment, not simply mirror the pooled dataset. For example, if you combine 150,000 archived desktop images with 5,000 smartphone images and then split randomly, the evaluation sets will be dominated by desktop images and will not properly reflect smartphone use.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Do Not Randomly Mix Sources When Building Validation and Test Sets
Use a Small Amount of Target-Distribution Data in Training with Auxiliary Data
When to Reduce the Influence of Extra Training Data
Training-Development Set
Comparing Errors Across Two Data Sources
Training–Dev Distribution Gap
Domain adaptation solves only certain distribution-shift problems
Choosing Development and Test Data for a Specific Application
Factory Photos and Inspection Tablet Images as a Distribution Mismatch Example
Random Train/Test Splits Can Be Misleading When Deployment Data Differs
Which data should define the dev and test sets for the plant-disease app?
Development and test examples should be drawn to match the kind of data the deployed system is expected to see.
Development and test sets should match the kind of data you expect later.
Match each data group for the plant-disease app with its role.
Order the dataset choices when training data and evaluation data come from different sources.
Why training data and evaluation data may come from different sources
Find the evaluation-set mismatch in a voice-command detector.
Why did strong desktop-photo results not predict mobile-upload success?
Choosing Training and Evaluation Data from Two Image Sources
If training uses extra web-sourced photos, the dev and test sets must also contain web-sourced photos.
Learn After
Why should source-mixed examples be kept out of both the development and test sets?
True or False: If your dataset mixes records from several sources with noticeably different distributions, the best practice is to randomly shuffle everything before creating the train, dev, and test splits.
For a movie-recommendation system, the dev and test sets should match the _____ of the users' future ratings, not a randomly shuffled archive.
If 188,000 website logs and 12,000 mobile-app logs are combined and then randomly split into dev/test sets, what fraction of the dev/test data will come from website logs?
If a model will be judged on roadside camera footage from one region, it is best to randomly mix together examples from several different regions when creating the dev and test sets.
Source-mix effect in a random split
Match each dev/test-set design choice to the effect it has on model development.
Put the reasoning steps in order for deciding whether mixed-source data should be randomly split into dev/test sets.
Selecting Validation and Test Data
If a dataset is formed by randomly mixing 4,000 photos from a mobile app with 156,000 photos collected from the web, the validation and test sets will reflect the mobile-app distribution.
Choose dev and test sets that match the data you expect to see in the _____.
Match each data-partitioning situation to the recommended outcome.
Order the steps for splitting mixed-source data so evaluation reflects the target distribution.
Analyze the consequences of mixing greenhouse and orchard photos before creating the dev and test sets.
Check a split strategy for a plant-disease app trained on mixed-source photos.
Why is a mixed dev/test split misleading in a warehouse-vision project?