Common Dev Set Sizes for Detecting 0.1% Improvements
Dev sets with 1,000 to 10,000 examples are common. With 10,000 examples, there is a good chance of detecting an improvement of 0.1%.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Small Dev Sets Cannot Detect Tiny Accuracy Differences
Common Dev Set Sizes for Detecting 0.1% Improvements
High-Value Applications May Need Larger Dev Sets
Statistical Significance Tests on Dev Set Changes
Which dev set best supports detecting a 0.1 percentage-point accuracy improvement?
A dev set should always be made as large as possible, even beyond what meaningful-change detection requires.
A dev set of _____ examples gives a good chance of detecting a 0.1% improvement.
Match each accuracy-detection goal with its dev set implication.
Order the reasoning process for sizing a dev set around a meaningful accuracy change.
Explain how the size of a meaningful accuracy improvement should influence dev set size.
Decide whether a mature recommendation system needs a larger dev set.
Why is a 100-example dev set inadequate for comparing 90.0% and 90.1% accuracy?
When is a dev set much larger than 10,000 examples most justified?
A team satisfied with detecting meaningful changes need not enlarge its dev set far beyond that requirement.
Learn After
How many dev set examples does Andrew Ng recommend to reliably detect a 0.1% accuracy improvement?
True or False: A dev set of 1,000 examples gives you a good chance of detecting a 0.1% accuracy improvement.
Dev sets with _____ to 10,000 examples are described as common in Machine Learning Yearning.
Which range represents the common size for a dev set according to Andrew Ng's Machine Learning Yearning?
True or False: A dev set of 10,000 examples gives you a good chance of detecting an accuracy improvement of 0.1%.
With _____ dev set examples, you have a good chance of detecting an accuracy improvement of 0.1%.
Match each dev set concept to its correct description.
Order the steps for reasoning about whether your dev set is large enough to detect a 0.1% improvement.
What accuracy improvement can you expect to reliably detect with a 10,000-example dev set, per Machine Learning Yearning?
True or False: A dev set of 1,000 examples is too small to fall within the commonly recommended dev set size range.
Dev sets with sizes from _____ to 10,000 examples are considered common in practice.
Match each dev set scenario to its correct outcome or classification.
Order the reasoning steps behind Andrew Ng's recommendation of 10,000 examples for detecting 0.1% improvements.
Explain the relationship between dev set size and the ability to detect minor accuracy improvements, citing common size ranges.
Evaluating a Dev Set Size for Detecting 0.1% Classifier Improvements
Determining the Dev Set Size for 0.1% Performance Changes