Learn Before
Choosing a Dev Set Large Enough to See Small Gains
A development set should contain enough examples to separate real model improvements from random fluctuation. For example, with only 100 labeled cases, it is difficult to tell whether a model at 88.4% accuracy is truly better than one at 88.5%. Many projects use a few hundred to a few thousand dev examples, and teams that need to detect very small gains may choose an even larger set.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why Dev and Test Sets Matter
How to Select Dev and Test Data for the Future Task
Choose Dev and Test Sets to Match the Main Goal When Feasible
Revise Evaluation Data and Metrics When They Stop Serving the Goal
Set the First Development and Test Splits Early for a New Project
Choosing a Dev Set Large Enough to See Small Gains
What is the main role of a development set during model iteration?
Another Name for the Development Set
Another name for the development set
Match each development-split role to its description.
Order the steps for choosing between two candidate spam filters using a validation set.
Which option is NOT a typical purpose of a development set?
The validation set is used to update model weights in the same way as the training set.
The validation set is used to tune parameters, _____ features, and make other choices about the learning method.
Match each development-set use to the task that illustrates it.
Arrange the main stages of a project that uses a development set.
What is the development set used for in model building?
Choosing the right split for model selection decisions.
What are the main roles of a development set?
Learn After
Tiny Development Sets Miss Very Small Accuracy Gains
Typical Development Set Sizes for Tiny Accuracy Gains
High-Impact Business Systems May Justify a Larger Development Set
Formal significance tests for validation-set changes
What dev set size is most suitable for spotting a 0.1 percentage-point gain in accuracy?
A development set should always be expanded to the maximum possible size, even after it is already large enough to reveal meaningful performance changes.
Validation set size for noticing a tiny accuracy change
Match each evaluation target with the dev set size it suggests.
Order the steps for deciding whether a dev set is large enough to detect a useful accuracy gain.
Match dev-set size to the smallest gain you care about.
Choose a dev set size that can detect a tiny but important gain.
Why is a 150-example dev set not enough to tell 83.0% from 83.4% accuracy?
When is a validation set much larger than 10,000 examples most justified?
If a validation set is already large enough to tell whether one model is meaningfully better than another, it does not need to be made much larger.