Tiny Development Sets Miss Very Small Accuracy Gains
A development set with 100 examples cannot reliably distinguish 88.0% accuracy from 88.1% accuracy. In many machine learning settings, a 100-example dev set is too small to measure a difference that small.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Tiny Development Sets Miss Very Small Accuracy Gains
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.
Learn After
What can a 100-example validation set tell you about 84.0% versus 84.1% accuracy?
Can a dev set with 100 examples reliably detect a 0.1 percentage-point accuracy gap?
Small Validation Sets Miss Tiny Accuracy Changes
Match each element in the comparison with its meaning.
Order the steps for judging whether a validation set can detect a small accuracy gap.
Limits of a 64-Example Development Set
Can a tiny validation set resolve a 0.1-point accuracy gap?
Why a 100-item validation set can miss a tiny gain
What is the clearest sign that the validation set is too small for this comparison?
A development set with 100 examples is usually small.