Resampling Rule and Curves for Reducing Variability
Question: If you repeatedly build many tiny training samples to make a noisy performance curve easier to read, what kind of resampling should be used for those samples, and what two mean error curves should be shown?
Sample answer: Those subsets should be formed by bootstrap-style resampling, so individual examples can appear more than once in the same subset. The two curves to plot are the mean training error and the mean development-set error.
Key points:
- Create the small subsets by resampling with replacement.
- Show the average training error and the average development-set error.
Rubric: The response must state that the subsets are drawn by resampling with replacement and must name both the mean training error and the mean development-set error (or average dev error) as the plotted quantities.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Decide First Whether Learning Curves Need Smoothing
What should you do separately for each randomly drawn subset when estimating a learning curve?
Averaging Repeated Runs to Smooth a Curve
After training separate models on many random samples of the same size, you plot the _____ training error and validation error across runs.
What is the main benefit of averaging dev-set learning curves from many randomly chosen training subsets?
The averaging method creates several smaller training subsets by sampling with replacement from the original data.
How Many Models Should Be Trained?
Match each term in the resampling method to its correct description.
Order the steps for averaging learning curves over multiple random mini-samples.
When estimating a learning curve by averaging over random subsets, how many subsets are usually sampled?
True or False: In the averaging method, one model is fit to the union of all randomly sampled training subsets, and that single model’s error is then averaged.
Errors recorded before averaging learning curves
Match each problem with the part of the averaging method that addresses it.
Arrange the steps for using repeated subset experiments to clarify a noisy learning curve.
Why do repeated random subsamples make learning curves less noisy, and how is the procedure carried out?
Estimating Curve Values from Repeated Resamples
Resampling Rule and Curves for Reducing Variability