Google

Developing learning algorithms that train on one distribution and generalize well to another is an important research problem. For making progress on a specific application rather than research progress, Machine Learning Yearning recommends choosing dev and test sets drawn from the same distribution.

Domain Adaptation for Different Data Distributions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

For progress on a specific machine learning application rather than research progress, dev and test sets should be chosen from the same distribution because this makes the team more efficient.

Application Progress Favors Same-Distribution Dev and Test Sets

What does ML Yearning recommend when your goal is progress on a specific application rather than research?

Domain adaptation methods are widely applicable and broadly used across most machine learning problems.

Domain adaptation involves training an algorithm on one _____ and having it generalize to a different one.

Match each concept from ML Yearning's domain adaptation discussion to its correct description.

Order the reasoning steps ML Yearning recommends when deciding how to handle differing data distributions in an application project.

How does ML Yearning characterize the scope of applicability of domain adaptation methods?

Choosing dev and test sets from the same distribution makes your ML team more efficient, according to ML Yearning.

ML Yearning recommends choosing dev and test sets drawn from the _____ distribution to efficiently make application progress.

Match each project goal to the strategy ML Yearning associates with it.

Order the key ideas in ML Yearning's argument for preferring same-distribution dev/test sets over domain adaptation in application work.

Question:
Discuss the difference in priorities when tackling domain adaptation as a research problem versus aiming to make progress on a specific machine learning application, according to Machine Learning Yearning.

Sample answer:
While developing algorithms that train on one distribution and generalize well to another (domain adaptation) is a valuable research pursuit, it is less efficient for practical applications. For a specific application, ML Yearning recommends choosing dev and test sets from the same distribution because domain adaptation methods are usually only applicable in special problem types.

Key points:
- Domain adaptation aims to train an algorithm on one distribution to generalize to another.
- It is an important research problem but typically applicable only to special problem types.
- For practical application progress, teams should choose dev and test sets from the same distribution.
- This same-distribution strategy increases team efficiency compared to attempting domain adaptation.

Rubric:
The response should contrast the broad goals of research on domain adaptation with the efficiency-focused strategy of using same-distribution dev and test sets for specific applications.

Contrast domain adaptation research with application-focused dataset strategies.

Case context:
Your startup is building a system to diagnose skin conditions from smartphone photos. You have a large dataset of clinical images (high quality) and a smaller dataset of user-submitted photos (varying quality). The engineering team suggests investing time in domain adaptation to train on clinical images and generalize to user photos.

Question:
Based on Machine Learning Yearning, should you adopt the engineering team's suggestion to focus on domain adaptation to achieve your immediate application goals? Explain your decision.

Sample answer:
No, you should not focus on domain adaptation. Because your goal is to make progress on a specific machine learning application, you should instead choose dev and test sets drawn from the same distribution (the user-submitted photos). Domain adaptation methods are usually applicable only in special problem types and focusing on same-distribution sets will make the team more efficient.

Key points:
- The goal is practical application progress, not general research.
- Domain adaptation is less widely used and often restricted to special problem types.
- The recommended approach is to draw dev and test sets from the same target distribution.
- Focusing on same-distribution sets improves team efficiency.

Rubric:
The response should advise against the domain adaptation approach for this specific application, citing the need for team efficiency and recommending same-distribution dev/test sets.

Evaluate data distribution strategies for a new health tech startup.

Question:
According to Machine Learning Yearning, why might a team prioritize choosing dev and test sets from the same distribution over trying to develop domain adaptation algorithms for a specific application?

Sample answer:
A team should prioritize same-distribution dev and test sets because it makes them more efficient. Domain adaptation is a complex research problem whose methods are typically applicable only in special cases, making it less suitable for straightforward application progress.

Key points:
- Choosing dev/test sets from the same distribution increases team efficiency.
- Domain adaptation is an important research problem but methods are less widely applicable.

Rubric:
The response must mention team efficiency and the specialized or research-oriented nature of domain adaptation.

Learn Before

Related