Why training data and evaluation data may come from different sources
Question: In a concise analytical response, explain why a dog detector might be trained with a mix of labeled web photos and storefront-camera images, while dev and test sets use only storefront-camera images.
Sample answer: The web photos can enlarge the training set and give the model more examples to learn from, which is helpful when labeled data are limited. However, the system will be deployed on storefront-camera images, so dev and test sets should match that future operating environment. Using storefront-camera data for evaluation makes the measured performance more relevant to the real task, even if the training data include additional sources. For that reason, the training distribution and the dev/test distribution can intentionally be different.
Key points:
- Web photos increase the amount of training data.
- Storefront-camera images match the deployment setting.
- Dev and test sets should reflect the data the model will face in production.
- Different distributions can be chosen on purpose when they serve different roles.
Rubric: A strong response explains that extra web photos help the model learn, states that storefront-camera images are the target distribution, ties dev/test selection to realistic evaluation, and notes that the training and evaluation distributions may differ by design.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Do Not Randomly Mix Sources When Building Validation and Test Sets
Use a Small Amount of Target-Distribution Data in Training with Auxiliary Data
When to Reduce the Influence of Extra Training Data
Training-Development Set
Comparing Errors Across Two Data Sources
Training–Dev Distribution Gap
Domain adaptation solves only certain distribution-shift problems
Choosing Development and Test Data for a Specific Application
Factory Photos and Inspection Tablet Images as a Distribution Mismatch Example
Random Train/Test Splits Can Be Misleading When Deployment Data Differs
Which data should define the dev and test sets for the plant-disease app?
Development and test examples should be drawn to match the kind of data the deployed system is expected to see.
Development and test sets should match the kind of data you expect later.
Match each data group for the plant-disease app with its role.
Order the dataset choices when training data and evaluation data come from different sources.
Why training data and evaluation data may come from different sources
Find the evaluation-set mismatch in a voice-command detector.
Why did strong desktop-photo results not predict mobile-upload success?
Choosing Training and Evaluation Data from Two Image Sources
If training uses extra web-sourced photos, the dev and test sets must also contain web-sourced photos.