Statistical Significance Tests on Dev Set Changes
One could test whether a change to an algorithm makes a statistically significant difference on the dev set. In practice, most teams do not do this unless they are publishing academic research papers, and the author usually does not find statistical significance tests useful for measuring interim progress.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Small Dev Sets Cannot Detect Tiny Accuracy Differences
Common Dev Set Sizes for Detecting 0.1% Improvements
High-Value Applications May Need Larger Dev Sets
Statistical Significance Tests on Dev Set Changes
Which dev set best supports detecting a 0.1 percentage-point accuracy improvement?
A dev set should always be made as large as possible, even beyond what meaningful-change detection requires.
A dev set of _____ examples gives a good chance of detecting a 0.1% improvement.
Match each accuracy-detection goal with its dev set implication.
Order the reasoning process for sizing a dev set around a meaningful accuracy change.
Explain how the size of a meaningful accuracy improvement should influence dev set size.
Decide whether a mature recommendation system needs a larger dev set.
Why is a 100-example dev set inadequate for comparing 90.0% and 90.1% accuracy?
When is a dev set much larger than 10,000 examples most justified?
A team satisfied with detecting meaningful changes need not enlarge its dev set far beyond that requirement.
Learn After
When are teams most likely to test whether a dev set change is statistically significant?
Most teams routinely test every dev set improvement for statistical significance.
Most teams skip significance tests unless publishing _____.
Match each context with the source's position on significance testing.
Order the reasoning for deciding whether to test a dev set change.
Explain the gap between theory and practice for significance tests on dev set changes.
Should a team test every interim dev set change for statistical significance?
Why are significance tests uncommon for interim dev set progress?
Which statement best captures the source's position on testing dev set changes?
The source rejects the possibility of testing dev set changes because teams rarely do it.