Why choose dev and test sets from the same distribution instead of building a special adaptation method?
Question: A machine learning team is deciding between two priorities: (1) making the dev and test sets match the data they care about most, or (2) investing time in a domain adaptation method for one particular deployment setting. According to the general strategy taught in machine learning practice, why is the first option often the better immediate choice?
Sample answer: Using dev and test sets drawn from the same distribution the team wants to improve on usually leads to faster progress. It gives clear feedback on whether a change helps the main task. By contrast, domain adaptation is a harder, research-heavy problem, and many techniques work only in limited situations, so it is not usually the best first route for everyday product improvement.
Key points:
- Same-distribution dev and test sets help the team move more efficiently.
- Domain adaptation is a research-oriented problem with limited generality.
Rubric: The response must mention improved team efficiency and the specialized, research-focused nature of domain adaptation.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Use Matching Dev and Test Distributions for Application Work
For an application project, what data-splitting rule is most helpful?
Domain adaptation methods are the default solution for most machine learning projects.
Domain adaptation means training on one _____ and expecting useful performance on another.
Match each machine-learning concept to its description.
Order the steps for choosing dev/test data when the project data differs from deployment data.
When are domain adaptation methods usually a good fit?
Selecting development and test sets from the same data distribution can improve team efficiency.
Dev and test sets should come from the _____ distribution.
Match each project goal to the strategy associated with it.
Order the main ideas behind matching dev and test data to the deployment setting.
Domain Adaptation Research Versus Application-Focused Data Splits
Choose evaluation strategy for a crop disease app.
Why choose dev and test sets from the same distribution instead of building a special adaptation method?