Learn Before
Dev and Test Splits Should Be Smaller for Very Large Datasets
A large holdout split is useful when data is limited, and an older rule of thumb was to reserve about 30% of the data for testing. That kind of split can be reasonable for small to medium datasets, such as those with roughly 100 to 10,000 examples. When datasets reach much larger scales, it is usually better to allocate a much smaller percentage to dev and test sets. The key requirement is that these sets be large enough to give a dependable estimate of performance, but no larger than needed for that purpose.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Dev and Test Splits Should Be Smaller for Very Large Datasets
A Test Set Large Enough for a Reliable Final Check
Test Set Size Should Enable Reliable Evaluation
Which use of a test set matches its intended role?
A final test set can measure performance without being used to choose model settings.
The held-out test set is used to _____ the final model’s performance on unseen data.
Match each holdout-set concept with its proper role.
Order the steps for deciding how a test set may be used.
What a test set is for, and what it is not for
Using the Right Evaluation Set for Model Selection
Test Set Purpose and Limits
Using the final holdout set appropriately
The test set should be used to choose the model you deploy.
Learn After
When Is a 70/30 Train/Test Split Reasonable?
True or False: If a project grows from 50,000 examples to 50 million examples, the dev and test sets must also grow by the same proportion to keep the split at 70/30.
In very large ML projects, the _____ of examples reserved for dev and test is often much smaller than the old 70/30 rule would suggest.
For what size of dataset is the simple 70/30 train/test split rule most suitable?
When a dataset grows to billions of examples, the percentage reserved for dev and test sets should also increase.
A 70/30 split is most appropriate when you have a _____ dataset, for example around 100 to 10,000 examples.
Connect each scale or idea to the best rule for choosing dev and test sets.
Order the steps for deciding dev/test set size when a dataset is very large.
Large datasets and the size of dev/test sets
A dev set should be large enough to estimate model performance reliably, but not larger than necessary for that purpose.
A common old rule was to reserve _____ of the data for testing, although that rule is not suitable for very large datasets.
Match each split-size concept to its meaning.
Order the actions a team should take when choosing test-set size as a dataset expands from thousands to tens of millions of examples.
How should holdout splits change when a dataset becomes very large?
Setting Holdout Sizes for a Very Large Image Dataset
How dev and test set sizes change when datasets become very large