Choosing a plot to judge whether more labeled data is worth the effort.
Case context: A team has trained a spam classifier on all 8,000 labeled emails currently available. Its training error and development error are known at that dataset size, but the team is considering spending several weeks labeling 4,000 additional emails.
Question: What graph should the team create, and which error curves should it include so they can decide whether collecting more data is likely to help?
Sample answer: The team should draw a learning curve that shows how both training error and development error change as the training set grows. For example, they could evaluate models trained on 1,000, 2,000, 4,000, 6,000, and 8,000 examples, then plot both curves on the same axes. Seeing the full pattern makes it easier to judge whether the development error is still dropping enough to justify labeling more data.
Key points:
- Create a learning curve over multiple training set sizes.
- Include both the training error curve and the development error curve.
- Use the development error trend to estimate whether more data is likely to help.
Rubric: The answer must recommend plotting a learning curve across several dataset sizes instead of relying on a single final measurement. It must also name both training error and development error as the curves to include.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Reading a High-Bias Learning Curve When the Train-Dev Gap Is Small
High Variance Clue from Learning Curves
Interpreting Training and Dev Error When Both Fit and Stability Are Poor
Why plot training error and development error on the same graph?
True or False: Looking only at the error from the largest training sample gives the full story of how performance changes as data increases.
Generalization Curves
Reading a Learning Curve
How to build and read a learning curve for a model
Why a full learning curve is more informative
Choosing a plot to judge whether more labeled data is worth the effort.
Why plot training and validation error together?
What does the far-right point on a learning curve indicate?
One data point is enough to understand how a model will change as more training examples are added.