Blackbox Dev Set
The Blackbox dev set is the second subset of a split dev set, consisting of the remaining examples after the Eyeball dev set is drawn; in the running example it contains 4,500 examples. It can be used to evaluate classifiers automatically by measuring their error rates, to select among algorithms, and to tune hyperparameters. One should avoid looking at it with one's eyes. The term Blackbox is used because this subset is only used to obtain blackbox evaluations of classifiers.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Eyeball Dev Set
Blackbox Dev Set
Using the Entire Dev Set as the Eyeball Dev Set When It Is Too Small to Split
Why explicitly divide a large dev set into examined and unexamined portions?
Does manual examination make one dev subset more prone to overfitting?
The unexamined dev subset can be used to tune _____.
Match each dev-set concept with its role or consequence.
Order the reasoning for handling a large dev set with many errors.
Explain how an explicit dev-set split exposes overfitting from manual error analysis.
How should a team handle about 1,000 errors in a 5,000-example dev set?
What distinct purpose does each portion of the split dev set serve?
Which development pattern signals overfitting to the manually inspected subset?
Should a dev set that is too small to divide still be split into two portions?
Learn After
Which of the following best describes the intended use of the Blackbox dev set?
True or False: It is appropriate to manually examine individual examples in the Blackbox dev set.
The Blackbox dev set is named after the _____ evaluations it provides, since examples in it are never manually inspected.
Which of the following is a valid use of the Blackbox dev set?
You should avoid looking at the Blackbox dev set with your eyes.
In the running example from Machine Learning Yearning, the Blackbox dev set contains _____ examples.
Match each dev set term to its defining characteristic.
Order the steps for setting up and using a split dev set with both Eyeball and Blackbox subsets.
What size range for a Blackbox dev set does Machine Learning Yearning say is sufficient for many applications?
The term 'Blackbox' is used because this dev set subset is used only for automated evaluations without manually examining examples.
The Blackbox dev set can be used to _____ among algorithms when comparing different model approaches.
Match each permitted use of the Blackbox dev set to a description of how that use is carried out.
Order the reasoning steps that justify keeping the Blackbox dev set off-limits for manual inspection.
Explain the core purpose and proper handling of a Blackbox dev set.
Diagnose a methodology error in a team's use of a split dev set.
Explain the origin of the term 'Blackbox' dev set.