Setting Holdout Sizes for a Very Large Image Dataset
Case context: A team is training an image classifier using 900 million labeled photos. A new engineer suggests reserving 30% of the data for evaluation because that is a common classroom rule of thumb.
Question: What is wrong with that proposal, and how should the team choose the fraction and absolute size of the dev/test sets?
Sample answer: That proposal is not appropriate for a dataset of this size. Keeping 30% of 900 million examples for evaluation would waste an enormous number of training examples. The team should use a much smaller share for the dev and test sets, far below 30%. The main rule is that these sets should be just large enough to give a dependable estimate of model performance; their size should be based on that need rather than on a fixed percentage.
Key points:
- A 30% evaluation split is far too large for 900 million examples
- The dev/test fractions should be much smaller than 30%
- The right size is the minimum needed for reliable performance measurement
Rubric: Response should correctly identify that a 30% split is inappropriate for a very large dataset. It must state that the dev/test fraction should be much smaller than 30%. It must specify that the holdout size is chosen by the amount needed to evaluate performance reliably rather than by a fixed percentage.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
When Is a 70/30 Train/Test Split Reasonable?
True or False: If a project grows from 50,000 examples to 50 million examples, the dev and test sets must also grow by the same proportion to keep the split at 70/30.
In very large ML projects, the _____ of examples reserved for dev and test is often much smaller than the old 70/30 rule would suggest.
For what size of dataset is the simple 70/30 train/test split rule most suitable?
When a dataset grows to billions of examples, the percentage reserved for dev and test sets should also increase.
A 70/30 split is most appropriate when you have a _____ dataset, for example around 100 to 10,000 examples.
Connect each scale or idea to the best rule for choosing dev and test sets.
Order the steps for deciding dev/test set size when a dataset is very large.
Large datasets and the size of dev/test sets
A dev set should be large enough to estimate model performance reliably, but not larger than necessary for that purpose.
A common old rule was to reserve _____ of the data for testing, although that rule is not suitable for very large datasets.
Match each split-size concept to its meaning.
Order the actions a team should take when choosing test-set size as a dataset expands from thousands to tens of millions of examples.
How should holdout splits change when a dataset becomes very large?
Setting Holdout Sizes for a Very Large Image Dataset
How dev and test set sizes change when datasets become very large