Choosing Development and Test Data for a Specific Application
Research on transfer across different data sources is important, but it is not usually the most direct way to make progress on one product. For an application project, it is often better to choose the development set and the test set from the same distribution, and to make that distribution match the data the system will face after deployment as closely as possible.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Do Not Randomly Mix Sources When Building Validation and Test Sets
Use a Small Amount of Target-Distribution Data in Training with Auxiliary Data
When to Reduce the Influence of Extra Training Data
Training-Development Set
Comparing Errors Across Two Data Sources
Training–Dev Distribution Gap
Domain adaptation solves only certain distribution-shift problems
Choosing Development and Test Data for a Specific Application
Factory Photos and Inspection Tablet Images as a Distribution Mismatch Example
Random Train/Test Splits Can Be Misleading When Deployment Data Differs
Which data should define the dev and test sets for the plant-disease app?
Development and test examples should be drawn to match the kind of data the deployed system is expected to see.
Development and test sets should match the kind of data you expect later.
Match each data group for the plant-disease app with its role.
Order the dataset choices when training data and evaluation data come from different sources.
Why training data and evaluation data may come from different sources
Find the evaluation-set mismatch in a voice-command detector.
Why did strong desktop-photo results not predict mobile-upload success?
Choosing Training and Evaluation Data from Two Image Sources
If training uses extra web-sourced photos, the dev and test sets must also contain web-sourced photos.
Learn After
Use Matching Dev and Test Distributions for Application Work
For an application project, what data-splitting rule is most helpful?
Domain adaptation methods are the default solution for most machine learning projects.
Domain adaptation means training on one _____ and expecting useful performance on another.
Match each machine-learning concept to its description.
Order the steps for choosing dev/test data when the project data differs from deployment data.
When are domain adaptation methods usually a good fit?
Selecting development and test sets from the same data distribution can improve team efficiency.
Dev and test sets should come from the _____ distribution.
Match each project goal to the strategy associated with it.
Order the main ideas behind matching dev and test data to the deployment setting.
Domain Adaptation Research Versus Application-Focused Data Splits
Choose evaluation strategy for a crop disease app.
Why choose dev and test sets from the same distribution instead of building a special adaptation method?