Google

Training data can differ from the data distribution one ultimately cares about. In the cat-app example, website images made up the training/test sets, while mobile-phone pictures were the actual distribution of interest; the algorithm did not generalize well to that distribution. Therefore, dev and test sets should be chosen to reflect the data one expects to get in the future and wants to do well on.

Training and Dev/Test Sets from Different Distributions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When available data sources differ from the distribution one cares about, randomly shuffling all available data into train/dev/test sets can make the dev/test sets fail to reflect the target distribution. In the cat-app example, shuffling user images together with many more internet images would make about 97.6% of dev/test data internet images, so it would not reflect the app-user distribution.

Avoid Randomly Shuffling Mixed-Source Data into Dev/Test Sets

Some data drawn from the same distribution as the dev/test set can be included in the training set. In the in-car audio example, if the training and training dev sets include in-car audio, good performance on in-car training data but not on in-car training dev data further validates the hypothesis that getting more in-car data would help.

Include Some Target-Distribution Examples in Training Alongside Auxiliary Data

When an additional training source has a very different distribution from the dev/test set, or when it is much larger than the target-distribution data, the auxiliary examples can be given lower weight. This can reduce the computational burden of making the model do well on both auxiliary and target-distribution examples.

Down-Weighting Auxiliary Data from a Different Distribution

A training dev set is a separate subset split from the available training data that the algorithm does not train on. It is drawn from the same distribution as the training set and only needs to be large enough to evaluate and track learning progress.

Training Dev Set

An error table for the cat image detector can put two data distributions on the x-axis and three error types on the y-axis: human-level error, error on examples the algorithm has trained on, and error on examples the algorithm has not trained on. Filling additional entries may sometimes give additional insight about what the algorithm is doing on the two data distributions.

Error Table Across Two Data Distributions and Three Error Types

Data mismatch occurs when an algorithm generalizes well to new data from the training-set distribution but not to data from the dev/test distribution. It is because the training set data is a poor match for the dev/test set data.

Data Mismatch Between Training and Dev Set Distributions

Domain adaptation is research on how to train an algorithm on one distribution and have it generalize to a different distribution. These methods are typically applicable only in special types of problems and are much less widely used than the ideas described in this chapter.

Limited Practical Scope of Domain Adaptation for Different Data Distributions

Developing learning algorithms that train on one distribution and generalize well to another is an important research problem. For making progress on a specific application rather than research progress, Machine Learning Yearning recommends choosing dev and test sets drawn from the same distribution.

Domain Adaptation for Different Data Distributions

In the cat app example, a classifier trained and tested on website images performs poorly after deployment because user-uploaded mobile-phone pictures have a different look, such as lower resolution, blurrier images, and poor lighting.

Website Images and Mobile Phone Pictures as a Distribution Mismatch Example

A random 70%/30% split into training and test sets can be a bad idea when the available training distribution differs from the distribution the system ultimately needs to perform well on.

Random 70/30 Train/Test Split Can Fail Under Distribution Shift

Which data should define the dev and test sets for the cat-picture app?

Dev and test sets should represent the future data distribution of interest.

Complete the principle: Dev and test sets should reflect _____ data.

Match each cat-app data group with its appropriate role or property.

Order the decisions for building datasets when auxiliary and target data differ.

Question:
In a concise analytical response, explain why a cat classifier might train on internet images plus mobile images while using only mobile images for dev and test sets.

Sample answer:
The additional internet images can help train a data-hungry algorithm by providing a much larger training set. However, user-uploaded mobile images closely reflect the actual distribution on which the app must perform well. Dev and test sets should therefore contain mobile images so that evaluation measures the desired future performance. This creates different training and dev/test distributions by design.

Key points:
- Internet images provide additional training data.
- Mobile images represent the actual distribution of interest.
- Dev and test sets should reflect expected future data.
- Training and dev/test distributions can intentionally differ.

Rubric:
A strong response identifies the value of auxiliary internet data, identifies mobile images as the target distribution, connects dev/test selection to future performance, and explains why the resulting distributions differ.

Explain why different training and evaluation distributions can be appropriate.

Case context:
A team trains a cat classifier using many downloaded internet images. Its dev and test sets also contain website images, but app users submit lower-resolution, blurrier, and poorly lit mobile-phone pictures. The classifier performs poorly on those uploads.

Question:
What should the team diagnose, and how should it revise its dev and test sets?

Sample answer:
The team should diagnose a mismatch between its website-image evaluation distribution and the mobile-image distribution it actually cares about. It should construct dev and test sets from user-uploaded mobile images because those reflect expected future inputs and the distribution on which the app should perform well. Internet images may still be used as additional training data.

Key points:
- Website images differ from mobile uploads.
- Current evaluation does not represent the target distribution.
- Dev and test sets should use mobile images.
- Internet images may remain useful for training.

Rubric:
The response should identify the distribution mismatch, select mobile images for dev/test, justify that choice using future target performance, and avoid claiming that internet images must be discarded from training.

Diagnose the evaluation-set mistake in a mobile cat-classification app.

Question:
Answer in one to three sentences using the distribution difference described in the cat-app example.

Sample answer:
Mobile uploads had a different look from website images, including being lower resolution, blurrier, and poorly lit. Because the training and test data consisted of website images, performance on those sets did not ensure generalization to the mobile distribution of interest.

Key points:
- Mobile and website images came from different distributions.
- Website-based evaluation did not establish performance on mobile uploads.

Rubric:
The answer must connect the visual differences in mobile images to the mismatch between the evaluated distribution and the target distribution.

Learn Before

Related