How dev and test set sizes change when datasets become very large
Question: When a machine learning project grows to a very large dataset, what happens to the share of examples reserved for dev/test sets, and what happens to the number of examples in those sets?
Sample answer: As the total dataset becomes very large, the share reserved for dev/test sets gets smaller, but the dev/test sets can still contain more examples in absolute terms because the overall dataset is much larger.
Key points:
- The share reserved for dev/test sets gets smaller.
- The absolute number of dev/test examples can still increase.
Rubric: The answer must state that the share of data reserved for dev/test sets gets smaller and the absolute number of examples in those sets can still increase.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
When Is a 70/30 Train/Test Split Reasonable?
True or False: If a project grows from 50,000 examples to 50 million examples, the dev and test sets must also grow by the same proportion to keep the split at 70/30.
In very large ML projects, the _____ of examples reserved for dev and test is often much smaller than the old 70/30 rule would suggest.
For what size of dataset is the simple 70/30 train/test split rule most suitable?
When a dataset grows to billions of examples, the percentage reserved for dev and test sets should also increase.
A 70/30 split is most appropriate when you have a _____ dataset, for example around 100 to 10,000 examples.
Connect each scale or idea to the best rule for choosing dev and test sets.
Order the steps for deciding dev/test set size when a dataset is very large.
Large datasets and the size of dev/test sets
A dev set should be large enough to estimate model performance reliably, but not larger than necessary for that purpose.
A common old rule was to reserve _____ of the data for testing, although that rule is not suitable for very large datasets.
Match each split-size concept to its meaning.
Order the actions a team should take when choosing test-set size as a dataset expands from thousands to tens of millions of examples.
How should holdout splits change when a dataset becomes very large?
Setting Holdout Sizes for a Very Large Image Dataset
How dev and test set sizes change when datasets become very large