Design a dev-set error learning curve experiment for an algorithm with 1,000 available training examples.
Case context: An engineer has accumulated a dataset of 1,000 labeled training examples. To diagnose whether the model would benefit from more data, the engineer decides to build a learning curve that plots dev-set error against training set size.
Question: Based on the provided concepts for constructing learning curves, describe how the engineer should set up the training runs and outline what the resulting plot should display.
Sample answer: The engineer should select several training-set sizes spanning up to the total of 1,000 examples, such as 100, 200, 300, and so on. They must train separate copies of the algorithm on each of these subset sizes. After training, the dev-set error is evaluated for each copy. The engineer then plots these dev-set error values on the vertical axis (y-axis) against the corresponding training-set sizes on the horizontal axis (x-axis) to create the learning curve.
Key points:
- Train separate copies of the algorithm on training subsets of varying sizes (e.g., 100, 200, 300, ..., 1,000).
- Evaluate the dev-set error for each of the trained algorithm copies.
- Plot the resulting dev-set error values against their respective training-set sizes.
Rubric: The response must detail: 1) selecting incremental subset sizes up to the maximum available (such as 100, 200, 300 up to 1,000), 2) training separate copies of the algorithm on these subsets, and 3) evaluating and plotting dev-set error versus the training-set sizes.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Nonlinear Training Set Sizes for Cheaper Learning Curves
What is plotted on the y-axis when constructing a learning curve by varying training set size?
When constructing a learning curve, you train a single model on the full dataset and evaluate it at regular checkpoints during training.
To plot a learning curve, you train _____ copies of the algorithm on training sets of different sizes.
Match each learning curve component to its role in the construction process.
Order the steps to construct a learning curve when 1,000 labeled training examples are available.
In the Machine Learning Yearning example with 1,000 training examples, which approach correctly constructs a learning curve?
When constructing a learning curve by training on subsets of 100, 200, and 300 examples, each model copy is evaluated on the same fixed dev set.
When constructing a learning curve, the x-axis represents _____ and the y-axis represents dev-set error.
Match each training subset description to its position or role on a learning curve built from 1,000 total examples.
Order the reasoning steps that explain why separate model copies must be trained for each subset size when constructing a learning curve.
Explain the methodology for constructing and plotting a dev-set error learning curve by varying training dataset size.
Design a dev-set error learning curve experiment for an algorithm with 1,000 available training examples.
How is dev-set error evaluated and plotted for different training-set sizes when constructing a learning curve?