How should holdout splits change when a dataset becomes very large?
Question: Describe why a fixed 70/30 train/test split is a poor choice for very large machine learning datasets. Compare a small dataset with a few hundred or a few thousand records to a massive dataset with tens of millions or more records. In your explanation, focus on how much data should be reserved for development and final evaluation, and why the role of those sets matters more than keeping a fixed percentage.
Sample answer: A 70/30 split can be practical for smaller projects because leaving out 30% still gives a training set that is large enough, while also providing enough holdout examples to judge performance. For very large datasets, though, keeping 30% aside would discard an enormous amount of useful training data without adding much evaluation value. The purpose of the development and test sets is to estimate how well a model generalizes and to compare model choices, not to consume a fixed share of the data. As the total dataset grows, the holdout fraction should become smaller, even though the actual number of examples in those sets can stay the same or increase a little so the evaluation remains reliable.
Key points:
- A 70/30 split can be reasonable when the dataset is small or moderate in size.
- With very large datasets, reserving 30% for holdout use wastes too many training examples.
- The share assigned to development and test sets should shrink as the dataset gets bigger.
- What matters is having enough holdout examples to measure performance accurately, not preserving a particular percentage.
Rubric: The response must explain: 1. Why a 70/30 split can work for smaller datasets. 2. Why keeping that same percentage is inefficient for very large datasets. 3. That development and test sets are meant to evaluate and compare models, so they only need enough examples to do that well, rather than a fixed large fraction of the data.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
When Is a 70/30 Train/Test Split Reasonable?
True or False: If a project grows from 50,000 examples to 50 million examples, the dev and test sets must also grow by the same proportion to keep the split at 70/30.
In very large ML projects, the _____ of examples reserved for dev and test is often much smaller than the old 70/30 rule would suggest.
For what size of dataset is the simple 70/30 train/test split rule most suitable?
When a dataset grows to billions of examples, the percentage reserved for dev and test sets should also increase.
A 70/30 split is most appropriate when you have a _____ dataset, for example around 100 to 10,000 examples.
Connect each scale or idea to the best rule for choosing dev and test sets.
Order the steps for deciding dev/test set size when a dataset is very large.
Large datasets and the size of dev/test sets
A dev set should be large enough to estimate model performance reliably, but not larger than necessary for that purpose.
A common old rule was to reserve _____ of the data for testing, although that rule is not suitable for very large datasets.
Match each split-size concept to its meaning.
Order the actions a team should take when choosing test-set size as a dataset expands from thousands to tens of millions of examples.
How should holdout splits change when a dataset becomes very large?
Setting Holdout Sizes for a Very Large Image Dataset
How dev and test set sizes change when datasets become very large