Small Dev Sets Cannot Detect Tiny Accuracy Differences
A dev set with 100 examples would not be able to detect a 0.1 percentage-point accuracy difference, such as 90.0% versus 90.1%. Compared to other machine learning problems the author has seen, a 100-example dev set is small.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Small Dev Sets Cannot Detect Tiny Accuracy Differences
Common Dev Set Sizes for Detecting 0.1% Improvements
High-Value Applications May Need Larger Dev Sets
Statistical Significance Tests on Dev Set Changes
Which dev set best supports detecting a 0.1 percentage-point accuracy improvement?
A dev set should always be made as large as possible, even beyond what meaningful-change detection requires.
A dev set of _____ examples gives a good chance of detecting a 0.1% improvement.
Match each accuracy-detection goal with its dev set implication.
Order the reasoning process for sizing a dev set around a meaningful accuracy change.
Explain how the size of a meaningful accuracy improvement should influence dev set size.
Decide whether a mature recommendation system needs a larger dev set.
Why is a 100-example dev set inadequate for comparing 90.0% and 90.1% accuracy?
When is a dev set much larger than 10,000 examples most justified?
A team satisfied with detecting meaningful changes need not enlarge its dev set far beyond that requirement.
Learn After
What can a 100-example dev set reveal about 90.0% versus 90.1% accuracy?
Can 100 dev examples detect a 0.1 percentage-point accuracy difference?
A 100-example dev set cannot detect a _____ percentage-point accuracy difference.
Match each element of the example to its source-based description.
Order the reasoning used to assess the 100-example dev set.
Explain why the example illustrates a limitation of a small dev set.
Diagnose whether a small dev set can distinguish two nearly equal classifiers.
Why is the 100-example dev set inadequate in the source example?
Which observation best supports calling the dev set too small for this comparison?
Does the author characterize 100 dev examples as small?