Case Study

Choosing a plot to judge whether more labeled data is worth the effort.

Case context: A team has trained a spam classifier on all 8,000 labeled emails currently available. Its training error and development error are known at that dataset size, but the team is considering spending several weeks labeling 4,000 additional emails.

Question: What graph should the team create, and which error curves should it include so they can decide whether collecting more data is likely to help?

Sample answer: The team should draw a learning curve that shows how both training error and development error change as the training set grows. For example, they could evaluate models trained on 1,000, 2,000, 4,000, 6,000, and 8,000 examples, then plot both curves on the same axes. Seeing the full pattern makes it easier to judge whether the development error is still dropping enough to justify labeling more data.

Key points:

  • Create a learning curve over multiple training set sizes.
  • Include both the training error curve and the development error curve.
  • Use the development error trend to estimate whether more data is likely to help.

Rubric: The answer must recommend plotting a learning curve across several dataset sizes instead of relying on a single final measurement. It must also name both training error and development error as the curves to include.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI