With separate training, training dev, dev, and test sets, one can evaluate training error on the training set, generalization to new data drawn from the training-set distribution on the training dev set, and performance on the target task using the dev and/or test sets.

Four Dataset Evaluation for Different Training and Dev/Test Distributions

Most guidelines for choosing the dev set size also apply to the training dev set.

Training Dev Set Size Guidelines

A training dev set is a separate subset split from the available training data that the algorithm does not train on. It is drawn from the same distribution as the training set and only needs to be large enough to evaluate and track learning progress.

Google

Training data can differ from the data distribution one ultimately cares about. In the cat-app example, website images made up the training/test sets, while mobile-phone pictures were the actual distribution of interest; the algorithm did not generalize well to that distribution. Therefore, dev and test sets should be chosen to reflect the data one expects to get in the future and wants to do well on.

Training and Dev/Test Sets from Different Distributions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When available data sources differ from the distribution one cares about, randomly shuffling all available data into train/dev/test sets can make the dev/test sets fail to reflect the target distribution. In the cat-app example, shuffling user images together with many more internet images would make about 97.6% of dev/test data internet images, so it would not reflect the app-user distribution.

Avoid Randomly Shuffling Mixed-Source Data into Dev/Test Sets

Some data drawn from the same distribution as the dev/test set can be included in the training set. In the in-car audio example, if the training and training dev sets include in-car audio, good performance on in-car training data but not on in-car training dev data further validates the hypothesis that getting more in-car data would help.

Include Some Target-Distribution Examples in Training Alongside Auxiliary Data

When an additional training source has a very different distribution from the dev/test set, or when it is much larger than the target-distribution data, the auxiliary examples can be given lower weight. This can reduce the computational burden of making the model do well on both auxiliary and target-distribution examples.

Down-Weighting Auxiliary Data from a Different Distribution

Training Dev Set

An error table for the cat image detector can put two data distributions on the x-axis and three error types on the y-axis: human-level error, error on examples the algorithm has trained on, and error on examples the algorithm has not trained on. Filling additional entries may sometimes give additional insight about what the algorithm is doing on the two data distributions.

Error Table Across Two Data Distributions and Three Error Types

Data mismatch occurs when an algorithm generalizes well to new data from the training-set distribution but not to data from the dev/test distribution. It is because the training set data is a poor match for the dev/test set data.

Data Mismatch Between Training and Dev Set Distributions

Domain adaptation is research on how to train an algorithm on one distribution and have it generalize to a different distribution. These methods are typically applicable only in special types of problems and are much less widely used than the ideas described in this chapter.

Limited Practical Scope of Domain Adaptation for Different Data Distributions

Developing learning algorithms that train on one distribution and generalize well to another is an important research problem. For making progress on a specific application rather than research progress, Machine Learning Yearning recommends choosing dev and test sets drawn from the same distribution.

Domain Adaptation for Different Data Distributions

In the cat app example, a classifier trained and tested on website images performs poorly after deployment because user-uploaded mobile-phone pictures have a different look, such as lower resolution, blurrier images, and poor lighting.

Website Images and Mobile Phone Pictures as a Distribution Mismatch Example

A random 70%/30% split into training and test sets can be a bad idea when the available training distribution differs from the distribution the system ultimately needs to perform well on.

Learn Before

Related

Learn After