Question:
What is the current relationship between the theoretical academic research on training and testing on different distributions and its practical application?

Sample answer:
There is still a huge gap between the theory of training and testing on different distributions and its practical application, making systematic academic study difficult.

Key points:
- There is a huge gap between theory and practice.
- Academic study is difficult to conduct systematically.

Rubric:
Full credit given for mentioning the "huge gap" between theory and practice.

Google

Domain adaptation is research on how to train an algorithm on one distribution and have it generalize to a different distribution. These methods are typically applicable only in special types of problems and are much less widely used than the ideas described in this chapter.

Limited Practical Scope of Domain Adaptation for Different Data Distributions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Applicability of Domain Adaptation

Gap Between Theory and Practice

Generalizing to Different Distributions _____

Concepts in Training on Different Distributions

The Challenge of Testing on Different Data

Question:
Explain what "luck" refers to when training an algorithm on dataset A and testing it on a very different dataset B, and discuss its impact on the academic study of this topic.

Sample answer:
When training an algorithm on dataset A and testing it on a completely different dataset B, "luck" refers to the researcher's hand-designed features tailored for the particular task, as well as other unknown or misunderstood factors. This reliance on luck can have a huge effect on how well the algorithm performs. Because the performance is so dependent on these unsystematic factors, it makes the academic study of training and testing on different distributions very difficult to carry out in a rigorous, systematic way.

Key points:
- Luck includes the researcher's hand-designed features.
- Luck includes other factors that are not yet understood.
- These factors can have a huge effect on algorithm performance on different data.
- This makes academic study difficult to carry out systematically.

Rubric:
A strong response will define luck in this context (hand-designed features, unknown factors), state that it heavily affects performance, and conclude that this makes academic study difficult to conduct systematically.

The Role of Luck in Different Distributions

Case context:
You are leading a machine learning team at an autonomous vehicle startup. Your team has trained an algorithm on a large dataset of sunny, daytime highway driving (Dataset A). They want to deploy it to navigate snowy, nighttime city streets (Dataset B), which represents a completely different data distribution. A junior engineer suggests using "domain adaptation" as a guaranteed, widely-used solution to seamlessly bridge this gap.

Question:
Based on Andrew Ng's advice, how should you evaluate the junior engineer's suggestion, and what practical realities should you consider regarding domain adaptation?

Sample answer:
I would advise the engineer to be cautious. While domain adaptation is the research area for generalizing from one distribution to another, there is still a huge gap between theory and practice. These methods are typically applicable only in special types of problems and are much less widely used than standard techniques. Furthermore, evaluating the algorithm on the snowy nighttime data (Dataset B) might rely heavily on "luck," such as our hand-designed features, making the system's performance unpredictable and difficult to study systematically.

Key points:
- There is a huge gap between domain adaptation theory and practice.
- Domain adaptation methods are typically applicable only in special types of problems.
- Performance on Dataset B could be hugely affected by luck and hand-designed features.

Rubric:
A comprehensive response should address the huge gap between theory and practice, note that domain adaptation has a limited scope (special problems), and mention the unpredictable effect of luck/hand-designed features.

Learn Before

Related