Some Data Should Be Left Out of Training
Examples that provide little or no useful signal should be excluded, especially when they do not match the distribution you care about at evaluation time. For example, if you are training a model to identify damaged highway signs, adding millions of unrelated office emails would waste computation and consume model capacity without helping the target task.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Bias (Informal Definition)
Variance, Informally
Some Data Should Be Left Out of Training
Mean Squared Error and the Bias-Variance Decomposition
Why Estimate the Lowest Plausible Error?
When Increasing Capacity Helps One Error and Hurts the Other
Validation-Error Learning Curve
Choosing Between Bias, Variance, and Dataset Mismatch Fixes
Diagnosing High Avoidable Bias from Similar Error Rates
A Single Model Can Show Several Error Sources at Once
A Big Gap Between Training and Development Error Suggests Overfitting
Reading High Bias from Training and Dev Error
A Model Can Show Both Bias and Variance
Strong Classification Performance Comes from Low Bias and Low Variance
What are the two main sources of error in machine learning?
Why can it be useful to tell whether a model's main problem is bias or variance before deciding how to improve it?
In the usual pair of major supervised-learning error sources, bias and ____ go together.
Which pair names the two broad error categories used to guide model improvement?
When a model's errors are due to bias or variance, that information helps you decide whether collecting more labels is likely to help.
Major error sources in supervised learning
Match Each Error-Analysis Idea to Its Role
Use bias and variance to choose an improvement strategy.
How can knowing the bias and variance pattern of your model help you make better decisions?
Sources of Error in Machine Learning Models
Knowing bias and variance helps you decide whether _____ is worth the effort.
Match each child concept to the bias-variance idea it supports.
Order the steps for deciding whether more labeled data is the right fix for a classifier.
Using Bias and Variance to Choose the Next Improvement
Choosing an Error Check Before Expanding the Dataset
Why Diagnose Bias and Variance Before Choosing an Improvement Strategy?
Some Data Should Be Left Out of Training
When Development and Test Sets Reflect Different Populations
How Model Capacity Changes the Risk of Mixing Data Sources
One Predictor Can Work Across Multiple Data Sources
Choose evaluation data to match the real-world target
Mismatched Auxiliary Data Source
Building Dev and Test Sets Before Real Users Exist
Refreshing Evaluation Sets After a Product Launch
Using Public Web Images When No Better Future-Like Data Exists
Judging How Much to Invest in Dev and Test Sets
What should determine dev and test set selection?
True or False: You can assume the training set and test set always come from the same distribution.
Development and test sets should reflect the conditions you expect after deployment, not only the _____ available in your training pool.
Why can a simple random test split be a poor choice when the data you expect in the future is different from the data you have now?
You can usually assume the data used for training and the data used for testing come from the same distribution.
Design Dev and Test Sets for the Future
Match each concept about development and test sets to its description.
Order the steps for choosing development and test sets when future data differs from training data.
What should dev and test examples be designed to resemble?
A validation and test set must exactly match the training distribution in every project.
Learn After
Why Skip Data That Does Not Match the Evaluation Set?
True or False: If the training, validation, and test sets are drawn from the same distribution, collecting more training examples will always make the model perform better.
According to machine learning strategy, if the dev error curve has _____, adding more training data is unlikely to help you reach your target.
Why might you exclude data that does not add useful information for training?
True or False: More training data always improves validation accuracy.
If the dev error curve has _____, adding more training data is unlikely to move you toward the target performance.
Match each machine-learning concept with the description that fits it best when deciding whether collecting more data is worthwhile.
Order the steps for using a learning curve to judge whether collecting more training data is worthwhile.
In a leaf-disease classifier, why should a large archive of handwritten invoices be left out of training?
True or False: Examining the learning curve can help you avoid spending months collecting more data that later turns out not to improve validation performance.
When compute is limited, examples that add no _____ should be left out of training.
Match each data scenario to the best action.
Order the steps for deciding whether to add a new data source to training.
Why Irrelevant Training Data Should Be Excluded
Whether to Add Contract Scans to a Plant Photo Classifier
Irrelevant Training Data and Model Capacity