Choosing a Development Set for Tiny Metric Gains
Question: A team wants enough validation data to notice a very small model gain, around 0.1%. About how large should the development set be?
Sample answer: A development set of about 10,000 examples is large enough to make that small change noticeable.
Key points:
- Use roughly 10,000 development examples.
- That size gives a reasonable chance of spotting a 0.1% improvement.
Rubric: The student's answer must state that a development set of 10,000 examples is needed to give a reasonable chance of detecting a 0.1% improvement.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What dev set size is commonly suggested for detecting a 0.1% improvement in accuracy?
True or False: A validation set with 1,000 examples is enough to reliably detect a 0.1% improvement in accuracy.
Development sets with _____ to 10,000 examples are often considered common.
Which dev set size range is commonly used in practical machine learning projects?
True or False: A dev set with 10,000 examples can often reveal an accuracy gain of about 0.1 percentage point.
Validation Set Size for Tiny Accuracy Gains
Match each development-set idea to its best description.
Put the validation-set sizing logic in order.
What size of accuracy change is a 10,000-example dev set generally able to detect?
True or False: A development set with 1,000 examples is smaller than the commonly suggested range for spotting very small improvements.
Common development-set sizes begin at _____ examples.
Match each development-set situation to the most accurate classification.
Arrange the steps for deciding when a dev set must be large enough to notice a tiny metric gain.
Why Dev Set Size Matters for Detecting Small Gains
Selecting a Dev Set Size for a Small Accuracy Gain
Choosing a Development Set for Tiny Metric Gains