How Should Dev and Test Sets Be Chosen for Product Work Versus General Research?
Question: In a practical machine learning product, how should the development and test sets be chosen, and how does that differ from a research setting where the goal is broader algorithmic progress?
Sample answer: For a practical product, the development and test sets should come from the same distribution so the team can compare models on data that reflects the target application. In a research setting, the emphasis is instead on building methods that can be trained on one distribution and still perform well when applied to another.
Key points:
- Product work calls for development and test sets from the same distribution.
- Research work focuses on methods that generalize from one distribution to another.
- Same-distribution development and test sets help the team make faster, more efficient decisions.
Rubric: The answer should state that practical product work requires development and test sets from the same distribution for efficiency, while research work focuses on algorithms that transfer well from one distribution to another.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
When the goal is to improve one deployed customer-support classifier, what should the development and test sets represent?
True or False: Training a model on one data distribution and getting it to work well on a different distribution is an important research challenge.
For reliable progress on one specific ML application, the development and test sets should come from the _____ distribution as the data the system will face in practice.
For an application-focused machine learning project, what dev/test setup is usually recommended?
True or False: For a focused machine learning product, using dev and test sets drawn from the same distribution usually helps the team move faster.
For an application to be judged fairly, the development and test sets should come from the _____ distribution.
Match each term to its best description in a discussion of how to choose dev and test sets.
Put the steps in order for choosing dev and test distributions in an application-driven ML project.
Which data-distribution issue is treated as a hard research challenge?
A development and test set drawn from the same distribution is equally emphasized for both application-focused progress and research-focused progress.
Using dev and test sets from the same distribution makes your _____ coordination more efficient.
Match each recommendation or outcome to the project goal it best supports.
Order the steps for choosing development and test distributions when planning a machine learning project.
Why matching the dev and test distributions can improve development speed
Aligning Development and Test Data for Faster Progress
How Should Dev and Test Sets Be Chosen for Product Work Versus General Research?