Learn Before
How to Select Dev and Test Data for the Future Task
Dev and test sets should be built from examples that match the data you expect to face after deployment, because those are the examples you want the model to do well on. A simple random split of the data you currently have can be misleading if future data will differ from the training examples. It is better to choose evaluation data based on the target use case than to assume the training and test distributions will naturally be the same.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why Dev and Test Sets Matter
How to Select Dev and Test Data for the Future Task
Choose Dev and Test Sets to Match the Main Goal When Feasible
Revise Evaluation Data and Metrics When They Stop Serving the Goal
Set the First Development and Test Splits Early for a New Project
Choosing a Dev Set Large Enough to See Small Gains
What is the main role of a development set during model iteration?
Another Name for the Development Set
Another name for the development set
Match each development-split role to its description.
Order the steps for choosing between two candidate spam filters using a validation set.
Which option is NOT a typical purpose of a development set?
The validation set is used to update model weights in the same way as the training set.
The validation set is used to tune parameters, _____ features, and make other choices about the learning method.
Match each development-set use to the task that illustrates it.
Arrange the main stages of a project that uses a development set.
What is the development set used for in model building?
Choosing the right split for model selection decisions.
What are the main roles of a development set?
Learn After
Some Data Should Be Left Out of Training
When Development and Test Sets Reflect Different Populations
How Model Capacity Changes the Risk of Mixing Data Sources
One Predictor Can Work Across Multiple Data Sources
Choose evaluation data to match the real-world target
Mismatched Auxiliary Data Source
Building Dev and Test Sets Before Real Users Exist
Refreshing Evaluation Sets After a Product Launch
Using Public Web Images When No Better Future-Like Data Exists
Judging How Much to Invest in Dev and Test Sets
What should determine dev and test set selection?
True or False: You can assume the training set and test set always come from the same distribution.
Development and test sets should reflect the conditions you expect after deployment, not only the _____ available in your training pool.
Why can a simple random test split be a poor choice when the data you expect in the future is different from the data you have now?
You can usually assume the data used for training and the data used for testing come from the same distribution.
Design Dev and Test Sets for the Future
Match each concept about development and test sets to its description.
Order the steps for choosing development and test sets when future data differs from training data.
What should dev and test examples be designed to resemble?
A validation and test set must exactly match the training distribution in every project.
How should a test set be chosen when deployment data will differ?
Match each data scenario to the best dev/test set choice.
Order the reasoning steps for deciding whether a dev/test split is appropriate.
Why a Random 30 Percent Split Can Be Misleading When Future Data Will Differ
Dev and Test Splits for a Field-Photo Classifier
How to Choose Dev and Test Data When Future Data Will Differ