Typical Development Set Sizes for Tiny Accuracy Gains
Development sets with about 1,000 to 10,000 examples are common. A set near 10,000 examples gives a reasonable chance of detecting a 0.1% improvement in accuracy.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Tiny Development Sets Miss Very Small Accuracy Gains
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.
Learn After
What dev set size is commonly suggested for detecting a 0.1% improvement in accuracy?
True or False: A validation set with 1,000 examples is enough to reliably detect a 0.1% improvement in accuracy.
Development sets with _____ to 10,000 examples are often considered common.
Which dev set size range is commonly used in practical machine learning projects?
True or False: A dev set with 10,000 examples can often reveal an accuracy gain of about 0.1 percentage point.
Validation Set Size for Tiny Accuracy Gains
Match each development-set idea to its best description.
Put the validation-set sizing logic in order.
What size of accuracy change is a 10,000-example dev set generally able to detect?
True or False: A development set with 1,000 examples is smaller than the commonly suggested range for spotting very small improvements.
Common development-set sizes begin at _____ examples.
Match each development-set situation to the most accurate classification.
Arrange the steps for deciding when a dev set must be large enough to notice a tiny metric gain.
Why Dev Set Size Matters for Detecting Small Gains
Selecting a Dev Set Size for a Small Accuracy Gain
Choosing a Development Set for Tiny Metric Gains