Estimating the Benefit of More Data
Case context: A recommendation team has graphed dev-set error as the number of labeled user sessions in training increases. They can describe the trend for the points they already have, but they are unsure how to estimate what would happen if the training set were expanded further.
Question: What is the main limitation in their analysis, and what additional plot should they create to improve it?
Sample answer: The analysis relies only on dev-set error, which is not easy to extend beyond the data already observed. The team should also plot training error, since comparing training and dev errors helps estimate how performance may change as more data is added.
Key points:
- The analysis uses only dev-set error.
- The main problem is that the dev-error trend is hard to project to larger training sizes.
- The team should add a training-error plot.
- Training error provides extra information for estimating the value of more data.
Rubric: A good response should identify that dev error alone is insufficient for projection, explain that extrapolating it is difficult, recommend adding training error, and connect that plot to estimating the effect of larger datasets.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
More Data Often Raises Training Error
Training Set Error Is Usually Lower Than Dev Set Error
Why plot training error when judging whether more data might improve a model?
A development-error curve computed by itself can be hard to project to much larger training sets.
The extra plot used to judge whether more data will help is _____.
Match each learning-curve element to its use when thinking about the value of additional data.
Arrange the steps for judging whether more data may help a model.
How does training error help assess the value of more data?
Estimating the Benefit of More Data
Why add a training-error curve?
What extra plot helps judge the value of collecting more data?
Training loss can help interpret a validation-loss curve when estimating the value of more data.