Essay

How should an evaluation set be sized when labeled data is abundant?

Question: Give a brief analytical explanation of what should determine the size of a human-reviewed development set when examples are easy to obtain. Also explain how the statement about 1,000 reviewed errors supports that idea.

Sample answer: When labeled data is plentiful, the main constraint is not how many examples exist, but how many cases people can realistically inspect by hand. The human-reviewed development set should therefore be sized according to practical review bandwidth and the time available for careful analysis. The note that it is unusual for someone to manually inspect more than 1,000 errors illustrates that this limit is operational rather than data-driven.

Key points:

  • Abundant data is not the binding constraint.
  • Human review capacity should guide the set size.
  • Reviewing more than 1,000 errors by hand is uncommon.
  • The 1,000-error statement describes a practical limit, not a minimum requirement.

Rubric: A strong response explains that manual inspection time is the key sizing factor, separates available data from review capacity, and interprets the 1,000-error remark as evidence of a practical limit rather than a target size.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI