Google

The Eyeball dev set is the portion of a split dev set that one manually examines for error analysis. It is named Eyeball as a reminder that it is being looked at with the eyes; for a speech recognition project in which one listens to audio clips, it might instead be called an Ear dev set. It is created by randomly selecting some fraction of the dev set, for example 10%, with the aim of including roughly 100 misclassified examples to look at. Continuing the running example, 10% of a 5,000-example dev set produces an Eyeball dev set of 500 examples, of which the algorithm is expected to misclassify about 100.

Eyeball Dev Set

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Because one gains intuition about the examples in the Eyeball dev set while looking at them, one will start to overfit the Eyeball dev set faster. If performance on the Eyeball dev set improves much more rapidly than performance on the Blackbox dev set, the Eyeball dev set has been overfit. Explicitly splitting the dev set into Eyeball and Blackbox subsets allows one to tell when the manual error analysis process is causing overfitting of the Eyeball portion.

Detecting Eyeball Dev Set Overfitting by Comparing Performance Against the Blackbox Dev Set

When access to data is plentiful, the Eyeball dev set size is determined mainly by how many examples one has time to manually analyze. It is rare for anyone to manually analyze more than 1,000 errors.

Sizing the Eyeball Dev Set When Data Is Plentiful

The Eyeball dev set should be large enough to give a sense of the algorithm's major error categories, especially for tasks that humans do well, such as recognizing cats in images.

Eyeball Dev Set Should Reveal Major Error Categories

If humans cannot do a task well, examining an Eyeball dev set may be less helpful because it is harder to determine why the algorithm misclassified an example; in that case, one might omit the Eyeball dev set.

Eyeball Dev Set May Be Omitted for Tasks Humans Cannot Do Well

What is the primary purpose of the Eyeball dev set when a large dev set is split into two subsets?

The name 'Eyeball dev set' is a reminder that a human manually looks at this portion of the dev set.

The Eyeball dev set is created by randomly selecting _____ of the dev set to be manually examined.

Match each term related to dev set splitting to its correct description.

Arrange the steps for creating and using an Eyeball dev set from a large dev set in the correct order.

If you take 10% of a 5,000-example dev set as your Eyeball dev set, how many examples does the Eyeball dev set contain?

A 500-example Eyeball dev set drawn from a 5,000-example dev set is expected to contain about 100 misclassified examples.

The Eyeball dev set should be large enough so that your algorithm misclassifies enough examples for you to _____.

Match each Eyeball dev set fact to the concept it illustrates.

Arrange the reasoning steps for deciding whether an Eyeball dev set is properly sized in the correct logical order.

Question:
When working on a speech recognition project instead of a computer vision project, explain how the name of the manually examined portion of the dev set changes, why it changes, and what the core purpose of this naming convention is.

Sample answer:
For a speech recognition project, the manually examined portion of the dev set is referred to as an 'Ear dev set' instead of an 'Eyeball dev set'. This name changes because the developer is listening to audio clips using their ears rather than looking at data with their eyes. The core purpose of using terms like 'Eyeball' or 'Ear' is to serve as a reminder that this specific subset is being manually examined by human senses for error analysis.

Key points:
- The subset is named an Ear dev set for speech recognition tasks.
- The name reflects the modality of manual examination, which is listening to audio clips.
- The naming convention acts as a reminder that the data is manually examined.

Rubric:
The response must identify the term 'Ear dev set', explain that the change is due to the auditory nature of speech recognition tasks (listening to audio clips), and explain that the naming convention serves as a reminder of manual human examination.

Explain how speech recognition tasks change the naming of the Eyeball dev set and the purpose of this naming convention.

Case context:
A machine learning team is working on a system with a 5,000-example dev set. They want to perform manual error analysis to identify major error categories and make progress. However, they also need to ensure that their final dev set performance evaluation remains completely objective and free from overfitting to the specific examples they manually inspect.

Question:
Based on this scenario and the principles of dev set splitting, describe the strategy the team should implement. Identify the two subsets they should create, explain their respective purposes, and state the rules regarding manual examination for each.

Sample answer:
The team should split their dev set into two subsets: an Eyeball dev set and a Blackbox dev set. The Eyeball dev set (e.g., 10% of the dev set) is designated for manual error analysis, allowing the team to look at misclassified examples. The Blackbox dev set is the remaining portion of the dev set and must not be manually examined, ensuring it remains a hands-off subset for objective evaluation.

Key points:
- Split the dev set into an Eyeball dev set and a Blackbox dev set.
- Use the Eyeball dev set for manual error analysis of misclassified examples.
- Keep the Blackbox dev set hands-off by avoiding any manual examination of its contents.

Rubric:
The response must propose splitting the dev set into an Eyeball dev set and a Blackbox dev set, specify that the Eyeball dev set is manually examined for error analysis, and specify that the Blackbox dev set must not be manually examined to maintain objective evaluation.

Propose a dev set splitting strategy to balance manual error analysis with objective performance evaluation.

Question:
What is the primary sizing requirement for an Eyeball dev set, and how many misclassified examples should it aim to contain for manual error analysis?

Sample answer:
The primary requirement is that the Eyeball dev set must be big enough so that the algorithm misclassifies enough examples to analyze. Specifically, it should aim to include roughly 100 misclassified examples for manual examination.

Key points:
- The Eyeball dev set must be large enough so that the algorithm misclassifies enough examples to analyze.
- The subset should aim to contain roughly 100 misclassified examples.

Rubric:
The answer should state that the Eyeball dev set must be large enough to contain enough misclassified examples for analysis, specifically targeting about 100 errors.

Learn Before

Related