Human-Review Dev Set
The human-review dev set is the portion of a split development set that people inspect directly for error analysis. The name reminds you that this subset is examined by a person rather than by the model alone; in an audio transcription project, a similar subset might be called an Ear dev set. A common way to build it is to randomly sample a small fraction of the dev set, such as 8% of a 6,250-example dev set, which yields 500 examples. If the system misclassifies about one-fifth of those examples, the human-review set would contain roughly 100 errors to study.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Human-Review Dev Set
Blackbox Dev Set
Use the Full Dev Set When It Is Too Small to Split
Why split a development set into a review subset and a tuning subset?
Does examining part of a dev set more closely increase the risk of overfitting to it?
The hands-off portion of the dev set can still be used to tune _____.
Match each dev-set idea with its role or consequence.
Order the reasoning for managing a dev set that is too large to inspect fully by hand.
Explain how separating a reviewed subset from an untouched subset can reveal overfitting during error analysis.
How should a team organize a large validation set with many mistakes?
How are the two parts of a split dev set used?
Which result suggests a model has been tuned too closely to the hand-checked subset?
If a development set is too small to divide into separate analysis and tuning subsets, using the full set for both purposes is a reasonable choice.
Learn After
Spotting Overfitting to a Manually Reviewed Dev Slice
How to Size an Eyeball Dev Set When Data Is Easy to Review
A Review Set Should Expose the Main Error Types
Manual Review Sets Can Be Unhelpful for Tasks People Cannot Judge Reliably
What is the main purpose of the human-review subset when a validation set is split into two parts?
The label "manual review set" can remind a team that people should inspect those examples directly.
What fraction of the dev set is manually reviewed in the Eyeball subset?
Match each dev set split term to its description.
Arrange the steps for building and using a small review subset from a larger development set in the correct order.
If 20% of a 3,000-example dev set is set aside as the Eyeball dev set, how many examples are in that subset?
If a 400-example Eyeball dev set has the same 20% error rate as the full dev set, you would expect about 80 misclassified examples.
The eye-check dev set should contain enough mistakes for you to _____.
Match each manual-review subset fact to the idea it describes.
Order the steps for deciding whether a small inspection subset is large enough for error analysis.
How should the manual-review subset be named in a voice transcription project?
Split the Development Set Into an Inspection Set and a Locked Set
How large should an Eyeball dev set be for useful manual error review?