Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
Question: Answer in one to three sentences: Why is a 150-example dev set not enough to distinguish classifiers with 83.0% and 83.4% accuracy?
Sample answer: The gap is only 0.4 percentage point, but each mistake on a 150-example set changes accuracy by about 0.67 percentage point. That makes the estimate too noisy to reliably tell those models apart.
Key points:
- The difference is only 0.4 percentage point.
- One mistake changes accuracy by about 0.67 point on 150 examples.
- A larger dev set is needed to compare them reliably.
Rubric: The answer must identify the small 0.4 percentage-point difference and state that 150 examples is too small, or too noisy, to detect it.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Tiny Development Sets Miss Very Small Accuracy Gains
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.