Google

When the dev set is large, for example 5,000 examples with a 20% error rate yielding about 1,000 misclassified images, manually examining all misclassified examples takes a long time, so one may decide not to use all of them for error analysis. In that case, the dev set can be explicitly split into two subsets: one that is manually looked at and one that is not. The portion that is manually examined will be overfit more rapidly, while the portion that is not manually looked at can be used to tune parameters. Splitting the dev set in this explicit way also lets one tell when the manual error analysis process is causing overfitting of the examined portion.

Splitting a Large Dev Set into a Manually Examined Subset and a Hands-Off Subset

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

The Eyeball dev set is the portion of a split dev set that one manually examines for error analysis. It is named Eyeball as a reminder that it is being looked at with the eyes; for a speech recognition project in which one listens to audio clips, it might instead be called an Ear dev set. It is created by randomly selecting some fraction of the dev set, for example 10%, with the aim of including roughly 100 misclassified examples to look at. Continuing the running example, 10% of a 5,000-example dev set produces an Eyeball dev set of 500 examples, of which the algorithm is expected to misclassify about 100.

Eyeball Dev Set

The Blackbox dev set is the second subset of a split dev set, consisting of the remaining examples after the Eyeball dev set is drawn; in the running example it contains 4,500 examples. It can be used to evaluate classifiers automatically by measuring their error rates, to select among algorithms, and to tune hyperparameters. One should avoid looking at it with one's eyes. The term Blackbox is used because this subset is only used to obtain blackbox evaluations of classifiers.

Blackbox Dev Set

If the dev set is not big enough to split, the entire dev set can be used as the Eyeball dev set for manual error analysis, model selection, and hyperparameter tuning.

Using the Entire Dev Set as the Eyeball Dev Set When It Is Too Small to Split

Why explicitly divide a large dev set into examined and unexamined portions?

Does manual examination make one dev subset more prone to overfitting?

The unexamined dev subset can be used to tune _____.

Match each dev-set concept with its role or consequence.

Order the reasoning for handling a large dev set with many errors.

Question:
In a concise analytical response, explain why repeatedly examining one portion can cause overfitting and how keeping another portion hands-off helps reveal it.

Sample answer:
Repeated manual inspection lets development decisions become increasingly tailored to the examined portion, so that portion is overfit more rapidly. By explicitly reserving another portion that is not manually examined, developers retain a separate basis for parameter tuning. Differences between results on the examined and unexamined portions can indicate that manual error analysis is causing overfitting to the examined data.

Key points:
- The manually examined portion is overfit more rapidly.
- The other portion is not manually examined.
- The unexamined portion can be used for parameter tuning.
- The explicit split helps reveal overfitting caused by manual error analysis.

Rubric:
A strong response identifies the greater overfitting risk for the examined portion, explains the role of the hands-off portion, and connects the explicit split to detecting overfitting caused by manual error analysis.

Explain how an explicit dev-set split exposes overfitting from manual error analysis.

Case context:
A team has a dev set of 5,000 examples with a 20% error rate, producing about 1,000 misclassified examples. Examining every error manually would take a long time.

Question:
What dev-set arrangement should the team choose, and how should it use each portion?

Sample answer:
The team should explicitly split the dev set into a portion it manually examines and a portion it does not examine. It should conduct error analysis on the examined portion and use the hands-off portion to tune parameters. Because the examined portion will be overfit more rapidly, comparing the portions can help the team detect overfitting caused by its manual error analysis.

Key points:
- Explicitly split the large dev set into two portions.
- Manually inspect only the examined portion.
- Keep the other portion free from manual inspection.
- Use the unexamined portion for parameter tuning.
- Use the separation to detect overfitting of the examined portion.

Rubric:
The response should recommend an explicit two-part split, assign manual error analysis to the examined portion, assign parameter tuning to the unexamined portion, and state how the split helps detect overfitting.

How should a team handle about 1,000 errors in a 5,000-example dev set?

Question:
Answer in one to three sentences, distinguishing the role of the manually examined portion from the role of the hands-off portion.

Sample answer:
The manually examined portion supports error analysis, though it will be overfit more rapidly. The hands-off portion can be used to tune parameters and helps show when manual error analysis has caused overfitting to the examined portion.

Key points:
- Examined portion: manual error analysis.
- Unexamined portion: parameter tuning.
- The split helps detect overfitting of the examined portion.

Rubric:
The answer should distinguish manual error analysis from hands-off parameter tuning and mention the examined portion's greater overfitting risk or the split's diagnostic value.

Learn Before

Related