Learn Before
Detecting Eyeball Dev Set Overfitting by Comparing Performance Against the Blackbox Dev Set
Because one gains intuition about the examples in the Eyeball dev set while looking at them, one will start to overfit the Eyeball dev set faster. If performance on the Eyeball dev set improves much more rapidly than performance on the Blackbox dev set, the Eyeball dev set has been overfit. Explicitly splitting the dev set into Eyeball and Blackbox subsets allows one to tell when the manual error analysis process is causing overfitting of the Eyeball portion.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Detecting Eyeball Dev Set Overfitting by Comparing Performance Against the Blackbox Dev Set
Sizing the Eyeball Dev Set When Data Is Plentiful
Eyeball Dev Set Should Reveal Major Error Categories
Eyeball Dev Set May Be Omitted for Tasks Humans Cannot Do Well
What is the primary purpose of the Eyeball dev set when a large dev set is split into two subsets?
The name 'Eyeball dev set' is a reminder that a human manually looks at this portion of the dev set.
The Eyeball dev set is created by randomly selecting _____ of the dev set to be manually examined.
Match each term related to dev set splitting to its correct description.
Arrange the steps for creating and using an Eyeball dev set from a large dev set in the correct order.
If you take 10% of a 5,000-example dev set as your Eyeball dev set, how many examples does the Eyeball dev set contain?
A 500-example Eyeball dev set drawn from a 5,000-example dev set is expected to contain about 100 misclassified examples.
The Eyeball dev set should be large enough so that your algorithm misclassifies enough examples for you to _____.
Match each Eyeball dev set fact to the concept it illustrates.
Arrange the reasoning steps for deciding whether an Eyeball dev set is properly sized in the correct logical order.
Learn After
Remedying an Overfit Eyeball Dev Set
What is the primary signal that your Eyeball dev set has been overfit during manual error analysis?
Manually examining Eyeball dev set examples causes you to overfit that set faster than if you had not examined them.
Explicitly splitting the dev set into Eyeball and Blackbox subsets allows you to detect when _____ is causing overfitting of the Eyeball portion.
Match each dev set concept to its correct description in the Eyeball/Blackbox framework.
Order the steps for detecting Eyeball dev set overfitting using the Blackbox dev set as a benchmark.
After several rounds of error analysis, your Eyeball dev set accuracy is 92% while your Blackbox dev set accuracy is 78%. What does this most likely indicate?
In the Eyeball/Blackbox framework, examples in the Blackbox dev set are regularly reviewed manually during error analysis.
If performance on the Eyeball dev set improves much more rapidly than on the Blackbox dev set, you have _____ the Eyeball dev set.
Match each observed performance pattern to its correct interpretation in the Eyeball/Blackbox framework.
Order the reasoning steps used to decide whether the Eyeball dev set has been overfit and what action to take.
Explain the mechanism of Eyeball dev set overfitting and how comparing it to a Blackbox dev set detects this issue.
Diagnosing divergent performance between Eyeball and Blackbox dev sets in a speech recognition system.
How does splitting the dev set help evaluate the manual error analysis process?