Applicability of Domain Adaptation

Gap Between Theory and Practice

Generalizing to Different Distributions _____

Concepts in Training on Different Distributions

The Challenge of Testing on Different Data

Question:
Explain what "luck" refers to when training an algorithm on dataset A and testing it on a very different dataset B, and discuss its impact on the academic study of this topic.

Sample answer:
When training an algorithm on dataset A and testing it on a completely different dataset B, "luck" refers to the researcher's hand-designed features tailored for the particular task, as well as other unknown or misunderstood factors. This reliance on luck can have a huge effect on how well the algorithm performs. Because the performance is so dependent on these unsystematic factors, it makes the academic study of training and testing on different distributions very difficult to carry out in a rigorous, systematic way.

Key points:
- Luck includes the researcher's hand-designed features.
- Luck includes other factors that are not yet understood.
- These factors can have a huge effect on algorithm performance on different data.
- This makes academic study difficult to carry out systematically.

Rubric:
A strong response will define luck in this context (hand-designed features, unknown factors), state that it heavily affects performance, and conclude that this makes academic study difficult to conduct systematically.

The Role of Luck in Different Distributions

Case context:
You are leading a machine learning team at an autonomous vehicle startup. Your team has trained an algorithm on a large dataset of sunny, daytime highway driving (Dataset A). They want to deploy it to navigate snowy, nighttime city streets (Dataset B), which represents a completely different data distribution. A junior engineer suggests using "domain adaptation" as a guaranteed, widely-used solution to seamlessly bridge this gap.

Question:
Based on Andrew Ng's advice, how should you evaluate the junior engineer's suggestion, and what practical realities should you consider regarding domain adaptation?

Sample answer:
I would advise the engineer to be cautious. While domain adaptation is the research area for generalizing from one distribution to another, there is still a huge gap between theory and practice. These methods are typically applicable only in special types of problems and are much less widely used than standard techniques. Furthermore, evaluating the algorithm on the snowy nighttime data (Dataset B) might rely heavily on "luck," such as our hand-designed features, making the system's performance unpredictable and difficult to study systematically.

Key points:
- There is a huge gap between domain adaptation theory and practice.
- Domain adaptation methods are typically applicable only in special types of problems.
- Performance on Dataset B could be hugely affected by luck and hand-designed features.

Rubric:
A comprehensive response should address the huge gap between theory and practice, note that domain adaptation has a limited scope (special problems), and mention the unpredictable effect of luck/hand-designed features.

Implementing Domain Adaptation in a Startup

Question:
What is the current relationship between the theoretical academic research on training and testing on different distributions and its practical application?

Sample answer:
There is still a huge gap between the theory of training and testing on different distributions and its practical application, making systematic academic study difficult.

Key points:
- There is a huge gap between theory and practice.
- Academic study is difficult to conduct systematically.

Rubric:
Full credit given for mentioning the "huge gap" between theory and practice.

Gap in Different Distribution Research

Factors Affecting Performance on Different Data

Widespread Use of Domain Adaptation

Domain adaptation is research on how to train an algorithm on one distribution and have it generalize to a different distribution. These methods are typically applicable only in special types of problems and are much less widely used than the ideas described in this chapter.

Google

Training data can differ from the data distribution one ultimately cares about. In the cat-app example, website images made up the training/test sets, while mobile-phone pictures were the actual distribution of interest; the algorithm did not generalize well to that distribution. Therefore, dev and test sets should be chosen to reflect the data one expects to get in the future and wants to do well on.

Training and Dev/Test Sets from Different Distributions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When available data sources differ from the distribution one cares about, randomly shuffling all available data into train/dev/test sets can make the dev/test sets fail to reflect the target distribution. In the cat-app example, shuffling user images together with many more internet images would make about 97.6% of dev/test data internet images, so it would not reflect the app-user distribution.

Avoid Randomly Shuffling Mixed-Source Data into Dev/Test Sets

Some data drawn from the same distribution as the dev/test set can be included in the training set. In the in-car audio example, if the training and training dev sets include in-car audio, good performance on in-car training data but not on in-car training dev data further validates the hypothesis that getting more in-car data would help.

Include Some Target-Distribution Examples in Training Alongside Auxiliary Data

When an additional training source has a very different distribution from the dev/test set, or when it is much larger than the target-distribution data, the auxiliary examples can be given lower weight. This can reduce the computational burden of making the model do well on both auxiliary and target-distribution examples.

Down-Weighting Auxiliary Data from a Different Distribution

A training dev set is a separate subset split from the available training data that the algorithm does not train on. It is drawn from the same distribution as the training set and only needs to be large enough to evaluate and track learning progress.

Training Dev Set

An error table for the cat image detector can put two data distributions on the x-axis and three error types on the y-axis: human-level error, error on examples the algorithm has trained on, and error on examples the algorithm has not trained on. Filling additional entries may sometimes give additional insight about what the algorithm is doing on the two data distributions.

Error Table Across Two Data Distributions and Three Error Types

Data mismatch occurs when an algorithm generalizes well to new data from the training-set distribution but not to data from the dev/test distribution. It is because the training set data is a poor match for the dev/test set data.

Data Mismatch Between Training and Dev Set Distributions

Limited Practical Scope of Domain Adaptation for Different Data Distributions

Developing learning algorithms that train on one distribution and generalize well to another is an important research problem. For making progress on a specific application rather than research progress, Machine Learning Yearning recommends choosing dev and test sets drawn from the same distribution.

Domain Adaptation for Different Data Distributions

In the cat app example, a classifier trained and tested on website images performs poorly after deployment because user-uploaded mobile-phone pictures have a different look, such as lower resolution, blurrier images, and poor lighting.

Website Images and Mobile Phone Pictures as a Distribution Mismatch Example

A random 70%/30% split into training and test sets can be a bad idea when the available training distribution differs from the distribution the system ultimately needs to perform well on.

Random 70/30 Train/Test Split Can Fail Under Distribution Shift

Which data should define the dev and test sets for the cat-picture app?

Dev and test sets should represent the future data distribution of interest.

Complete the principle: Dev and test sets should reflect _____ data.

Match each cat-app data group with its appropriate role or property.

Order the decisions for building datasets when auxiliary and target data differ.

Question:
In a concise analytical response, explain why a cat classifier might train on internet images plus mobile images while using only mobile images for dev and test sets.

Sample answer:
The additional internet images can help train a data-hungry algorithm by providing a much larger training set. However, user-uploaded mobile images closely reflect the actual distribution on which the app must perform well. Dev and test sets should therefore contain mobile images so that evaluation measures the desired future performance. This creates different training and dev/test distributions by design.

Key points:
- Internet images provide additional training data.
- Mobile images represent the actual distribution of interest.
- Dev and test sets should reflect expected future data.
- Training and dev/test distributions can intentionally differ.

Rubric:
A strong response identifies the value of auxiliary internet data, identifies mobile images as the target distribution, connects dev/test selection to future performance, and explains why the resulting distributions differ.

Explain why different training and evaluation distributions can be appropriate.

Case context:
A team trains a cat classifier using many downloaded internet images. Its dev and test sets also contain website images, but app users submit lower-resolution, blurrier, and poorly lit mobile-phone pictures. The classifier performs poorly on those uploads.

Question:
What should the team diagnose, and how should it revise its dev and test sets?

Sample answer:
The team should diagnose a mismatch between its website-image evaluation distribution and the mobile-image distribution it actually cares about. It should construct dev and test sets from user-uploaded mobile images because those reflect expected future inputs and the distribution on which the app should perform well. Internet images may still be used as additional training data.

Key points:
- Website images differ from mobile uploads.
- Current evaluation does not represent the target distribution.
- Dev and test sets should use mobile images.
- Internet images may remain useful for training.

Rubric:
The response should identify the distribution mismatch, select mobile images for dev/test, justify that choice using future target performance, and avoid claiming that internet images must be discarded from training.

Diagnose the evaluation-set mistake in a mobile cat-classification app.

Question:
Answer in one to three sentences using the distribution difference described in the cat-app example.

Sample answer:
Mobile uploads had a different look from website images, including being lower resolution, blurrier, and poorly lit. Because the training and test data consisted of website images, performance on those sets did not ensure generalization to the mobile distribution of interest.

Key points:
- Mobile and website images came from different distributions.
- Website-based evaluation did not establish performance on mobile uploads.

Rubric:
The answer must connect the visual differences in mobile images to the mismatch between the evaluated distribution and the target distribution.

Learn Before

Related

Learn After