Google

Dev and test sets should be chosen to reflect data one expects to receive in the future and wants to perform well on. The test set should not simply be 30% of the available data when future data is expected to differ from the training data. One should not assume the training distribution is the same as the test distribution; examples should reflect what one ultimately wants to perform well on rather than only the data available for training.

Choosing Dev and Test Sets to Reflect Future Data

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Data with no benefit should be left out for computational reasons. In the cat-detector example, scanned historical documents that contain nothing resembling a cat and look completely unlike the dev/test distribution have negligible benefit, and including them would waste computation resources and neural-network representation capacity.

Adding More Training Data Does Not Always Help

Mismatched dev and test sets introduce uncertainty about whether improving on the dev-set distribution will also improve test-set performance, making it harder to determine what is working and to prioritize what to work on.

Special Challenges from Different Training and Dev/Test Distributions

Merging user-uploaded images with internet images had a real risk of worsening performance for earlier learning algorithms, such as hand-designed computer-vision features followed by a simple linear classifier. With powerful, flexible algorithms such as large neural networks, that risk has greatly diminished.

Risk of Merging Training Data Sources Depends on Algorithm Flexibility

The case for combining internet images with mobile-app images relies on an x-to-y mapping that works well for both types of data. Such a system can input either an internet image or a mobile-app image and reliably predict the label without knowing the image source.

Shared Label Mapping Across Data Sources

Training data can differ from the data distribution one ultimately cares about. In the cat-app example, website images made up the training/test sets, while mobile-phone pictures were the actual distribution of interest; the algorithm did not generalize well to that distribution. Therefore, dev and test sets should be chosen to reflect the data one expects to get in the future and wants to do well on.

Training and Dev/Test Sets from Different Distributions

An auxiliary data source is inconsistent with the target task when the same input features can imply different labels depending on the data source. If one only wants to predict New York City housing prices, Detroit housing data is inconsistent because the same house size can have a very different price in the two cities, so mixing the datasets would hurt performance.

Inconsistent Auxiliary Data Source

Before a mobile app has launched, actual user data may not yet be available for dev/test sets. In that case, one can try to approximate the future data, for example by asking friends to send mobile-phone cat pictures.

Approximating Future Dev/Test Data Before Launch

After a mobile app is launched, the dev and test sets can be updated using actual user data.

Updating Dev/Test Sets with Actual User Data After Launch

If there is no way to get data that approximates expected future data, one can start with website images, but should be aware of the risk that this can lead to a system that does not generalize well.

Risk of Starting with Website Images When Future-Like Data Is Unavailable

Deciding how much to invest in developing great dev and test sets requires judgment.

Development Investment for Dev and Test Sets Requires Judgment

According to Machine Learning Yearning, what is the primary criterion for choosing dev and test sets?

True or False: When building a dev/test set, it is safe to assume the training distribution is the same as the test distribution.

Dev and test sets should contain examples that reflect what you ultimately want to perform well on, rather than only the _____ you happen to have for training.

Why is using a simple 30% random split of available data as your test set problematic when future data differs from training data?

According to ML Yearning, it is generally safe to assume your training data distribution is the same as your test data distribution.

Dev and test sets should be chosen to reflect data you expect to get in the _____ and want to do well on.

Match each dev/test set concept from ML Yearning to its correct description.

Order the steps for correctly choosing dev and test sets according to ML Yearning's guidance.

According to ML Yearning, what should the examples in your dev and test sets primarily reflect?

According to ML Yearning, dev and test sets must always come from the same distribution as the training data.

ML Yearning warns that the test set should not simply be _____ of the available data when future data differs from the training set.

Match each data scenario to the correct dev/test set strategy decision according to ML Yearning.

Order the reasoning steps for deciding whether a proposed dev/test set is well-chosen, per ML Yearning.

Question:
Explain why simply designating 30% of your available training data as a test set is problematic when you expect future data to differ in nature from your training data. Describe how you should choose your dev and test sets under these conditions according to Machine Learning Yearning.

Sample answer:
A standard 30% split assumes the training distribution is the same as the test distribution. If future data differs from training data, the resulting test set will only reflect the training distribution rather than what the system expects to receive in the future. According to Machine Learning Yearning, dev and test sets should be chosen to reflect the data one expects to get in the future and wants to perform well on, rather than simply whatever data happens to be available for training.

Key points:
- A 30% random split fails to capture differing future distributions.
- Training and test distributions should not be assumed to be the same.
- Dev and test sets must reflect future data that the system needs to perform well on.

Rubric:
The answer should explain that a 30% split fails because it assumes the training and test distributions are identical, and explain that dev/test sets must reflect the future data one expects to get and wants to perform well on.

Why Standard Data Splits Fail With Different Future Distributions

Case context:
A team is building a cat detector app. They have collected 100,000 website images for training. However, the app will run on mobile devices, and the team expects the future data from mobile phone images to differ in nature from the website images. The team decides to take 30% of the website images to serve as their dev and test sets.

Question:
According to Machine Learning Yearning, diagnose the problem with the team's split strategy and explain how they should choose their dev and test sets instead.

Sample answer:
The problem is that the dev and test sets are taken from the website images distribution, which differs from the mobile phone images distribution expected in the future. The team should not assume the training distribution is the same as the test distribution, nor should they simply split 30% of the available training data. Instead, they should choose dev and test sets that reflect the mobile phone images they expect to receive in the future and want the system to perform well on.

Key points:
- A 30% random split of website images does not represent the mobile phone image distribution.
- Dev and test sets should be chosen to reflect future data (mobile phone images) rather than training data (website images).
- The team must design the evaluation sets around the distribution they want to perform well on.

Rubric:
The answer must identify that a 30% split of website images is problematic because mobile phone images differ in distribution, and state that dev/test sets must reflect the future mobile phone image distribution the system is targeted to perform well on.

Dev and Test Set Design for Mobile Image Applications

Question:
When selecting examples for dev and test sets, what is the core criterion you should follow if you expect your future data to differ from your training data?

Sample answer:
Dev and test sets should be chosen to reflect the data you expect to get in the future and want to do well on, rather than the distribution of the training data.

Key points:
- Dev and test sets must reflect future data.
- Selection must align with the target distribution the system wants to perform well on.

Rubric:
The answer must state that the sets must reflect future data that the system expects to receive and wants to perform well on.

Learn Before

Related