How should an evaluation set be sized when labeled data is abundant?
Question: Give a brief analytical explanation of what should determine the size of a human-reviewed development set when examples are easy to obtain. Also explain how the statement about 1,000 reviewed errors supports that idea.
Sample answer: When labeled data is plentiful, the main constraint is not how many examples exist, but how many cases people can realistically inspect by hand. The human-reviewed development set should therefore be sized according to practical review bandwidth and the time available for careful analysis. The note that it is unusual for someone to manually inspect more than 1,000 errors illustrates that this limit is operational rather than data-driven.
Key points:
- Abundant data is not the binding constraint.
- Human review capacity should guide the set size.
- Reviewing more than 1,000 errors by hand is uncommon.
- The 1,000-error statement describes a practical limit, not a minimum requirement.
Rubric: A strong response explains that manual inspection time is the key sizing factor, separates available data from review capacity, and interprets the 1,000-error remark as evidence of a practical limit rather than a target size.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What usually limits the size of a manual review set when data is abundant?
Eyeball Dev Set Size Under Abundant Data
When data is abundant, the human-reviewed development set should be sized according to how many examples the team can inspect _____.
Link each dev-set sizing idea to its meaning.
Order the steps for choosing a development set when data are plentiful.
How should an evaluation set be sized when labeled data is abundant?
How should a manual-audit sample be sized when records are plentiful?
What does it suggest when a team has manually examined more than 1,000 mistakes?
Which choice best matches the advice for sizing a review set when data is plentiful?
True or false: A recommendation of analyzing about 1,000 mistakes means the Eyeball dev set must contain exactly 1,000 examples.