Learn Before
Using a learning curve to judge whether more data will help
Case context: You are building a fraud detection model for an online marketplace, and its development-set error is still too high. You are considering whether collecting several more months of labeled transaction data is worth the time and cost.
Question: How can a learning curve help you decide whether more training data is likely to improve performance?
Sample answer: Plot the development-set error for several training-set sizes taken from your current data. Then examine how the error changes as the training set gets larger. If the curve is still dropping, the model may benefit from additional labeled examples. If the curve has become nearly flat, more data is unlikely to produce a large improvement.
Key points:
- Plot development-set error against training-set size
- Check how the error changes as more examples are added
- Use the trend to decide whether gathering more data is worthwhile
Rubric: The answer should describe plotting development-set error versus the number of training examples, using the trend to judge whether additional data is likely to help.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Constructing a Performance Curve from Different Training Set Sizes
Larger Training Sets Usually Lower Dev-Set Error
Target Error Rate on a Learning Curve
Using a Development-Set Curve to Estimate the Payoff of More Data
Using Training Error to Judge the Value of More Data
Why plot training and development error together on a learning curve?
Tiny Training Samples Make Learning Curves Unstable
What are the axes of a learning curve?
A learning curve shows error versus network depth.
Learning Curves and Validation Error
Parts of a learning curve
Constructing a learning curve for model selection
What a learning curve shows
Using a learning curve to judge whether more data will help
What a Learning Curve Shows
What belongs on the vertical axis of a learning curve?
A learning curve is less informative than a single validation score.