Learn Before
Choosing a Dev Set Large Enough to See Small Gains
Choose the development-set size according to the smallest performance gain that must influence model selection. For ordinary unweighted accuracy on 100 examples, each additional correct prediction changes measured accuracy by 1 percentage point, so the set cannot directly register a 0.1-percentage-point difference such as 88.0% versus 88.1%. Detecting smaller meaningful gains generally requires a larger development set, while further expansion is unnecessary once the set can reliably resolve the differences that matter.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why Dev and Test Sets Matter
How to Select Dev and Test Data for the Future Task
Choose Dev and Test Sets to Match the Main Goal When Feasible
Revise Evaluation Data and Metrics When They Stop Serving the Goal
Set the First Development and Test Splits Early for a New Project
Choosing a Dev Set Large Enough to See Small Gains
What is the main role of a development set during model iteration?
Another Name for the Development Set
Another name for the development set
Match each development-split role to its description.
Order the steps for choosing between two candidate spam filters using a validation set.
Which option is NOT a typical purpose of a development set?
The validation set is used to update model weights in the same way as the training set.
The validation set is used to tune parameters, _____ features, and make other choices about the learning method.
Match each development-set use to the task that illustrates it.
Arrange the main stages of a project that uses a development set.
What is the development set used for in model building?
Choosing the right split for model selection decisions.
What are the main roles of a development set?
Learn After
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.
Sizing a Training Development Set
Accuracy Resolution on a 100-Example Development Set