Blackbox Dev Set
The Blackbox dev set is the portion of a split development set that remains after an Eyeball dev set has been separated out. In one example, it contains 3,800 examples. It is used only for automated evaluation: you can measure error rates, compare models, and tune hyperparameters, but you should not inspect individual examples manually. It is called Blackbox because it is reserved for nonvisual, automated checks rather than direct review.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Human-Review Dev Set
Blackbox Dev Set
Use the Full Dev Set When It Is Too Small to Split
Why split a development set into a review subset and a tuning subset?
Does examining part of a dev set more closely increase the risk of overfitting to it?
The hands-off portion of the dev set can still be used to tune _____.
Match each dev-set idea with its role or consequence.
Order the reasoning for managing a dev set that is too large to inspect fully by hand.
Explain how separating a reviewed subset from an untouched subset can reveal overfitting during error analysis.
How should a team organize a large validation set with many mistakes?
How are the two parts of a split dev set used?
Which result suggests a model has been tuned too closely to the hand-checked subset?
If a development set is too small to divide into separate analysis and tuning subsets, using the full set for both purposes is a reasonable choice.
Learn After
What is the main role of a black-box validation set in model development?
True or False: The Blackbox dev set is meant for manually checking individual examples.
An evaluation set that is kept out of manual review during development is often called a _____ set.
Which is an appropriate use of a Blackbox dev set?
A black-box development set should be used only for automated evaluation, not for manual inspection.
A validation split example
Match each evaluation-set term to its definition.
Order the steps for splitting a validation pool into a review set and a machine-only set.
How large is a Blackbox dev set often recommended for many tasks?
A blackbox dev set is evaluated only by automatic metrics, not by reviewing individual examples.
A validation set can be used to _____ candidate algorithms before deployment.
Match each valid purpose of an automated holdout set to the way it is used.
Order the steps that explain why an untouched hidden validation set should stay hidden.
Describe what a blackbox dev set is and how it should be used.
Find the mistake in how a team uses its split evaluation set.
Why is this evaluation subset called the black-box set?