Formal significance tests for validation-set changes
A model update can be checked to see whether its change in validation performance is statistically significant. In day-to-day machine learning work, most teams do not run a formal test for every small improvement. Those tests are used more often in research settings than in routine product development, and many teams rely instead on the size and consistency of the observed gain.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Tiny Development Sets Miss Very Small Accuracy Gains
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.
Learn After
In which situation are teams most likely to check whether a dev-set improvement is statistically significant?
Most teams formally test every validation-set improvement for statistical significance.
Most teams skip significance tests unless submitting a _____.
Match each setting with the view on statistical significance checks.
Order the reasoning for deciding whether to run a significance test on a validation-set change.
Explain why statistical tests are usually not a day-to-day tool for validation-set changes.
Should interim dev-set comparisons always include a statistical test?
Why are formal significance checks rarely used during dev-set iteration?
Which statement best reflects the guidance on statistical tests for development-set changes?
True or False: There is no statistical method for comparing two development-set results.