Why a 100-item validation set can miss a tiny gain
Question: Answer in one to three sentences using the accuracy figures and the stated limitation.
Sample answer: A 100-example validation set is too coarse to tell whether a model at 84.0% is really better than one at 84.6%, because one example changes the score by 1 percentage point. With so few cases, the observed gap could just be noise rather than a real improvement.
Key points:
- 100-example validation set
- 84.0% versus 84.6% accuracy
- The difference is too small to measure reliably
Rubric: The answer should refer to the 100-example size, the two accuracy values or a sub-1-point difference, and the fact that this gap cannot be distinguished reliably.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
What can a 100-example validation set tell you about 84.0% versus 84.1% accuracy?
Can a dev set with 100 examples reliably detect a 0.1 percentage-point accuracy gap?
Small Validation Sets Miss Tiny Accuracy Changes
Match each element in the comparison with its meaning.
Order the steps for judging whether a validation set can detect a small accuracy gap.
Limits of a 64-Example Development Set
Can a tiny validation set resolve a 0.1-point accuracy gap?
Why a 100-item validation set can miss a tiny gain
What is the clearest sign that the validation set is too small for this comparison?
A development set with 100 examples is usually small.