Learn Before
Interpreting Training Error on a Large Dataset
Case context: A developer fits a neural network on 80,000 labeled records and finds that it still misclassifies 12% of the training examples.
Question: In the informal sense used in the course material, what does this 12% training error indicate, and why does the size of the training set matter for that interpretation?
Sample answer: The 12% training error is a practical estimate of the model’s bias. In this informal usage, bias means the model’s error on the training data. Because the training set is very large, that error rate is a reasonable stand-in for bias rather than a number that is heavily distorted by a tiny sample.
Key points:
- The 12% error on the training set is taken as an estimate of bias.
- A very large training set makes the training error a useful approximation of bias.
Rubric: The response must identify the 12% training error rate as bias under the informal definition, and note that a very large training set is what makes the training error a good proxy for bias.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
High Training Error Calls for Better Fit, Not More Data
Low Bias and Low Variance Mean Good Performance
A Negative Gap to the Best Achievable Error Usually Signals Memorization
Informal and Technical Uses of Bias and Variance
Training Performance Is a Baseline for Held-Out Performance
Informal Meaning of Model Bias
True or False: In this framework, the term bias refers to a model’s error on the validation or test set.
Informal meaning of bias in a large-sample setting
Informal Meaning of Bias
In this informal usage, bias means the model's error on the training set.
Informally, an algorithm's _____ is the error rate on the training set.
Match each learning-curve term to its meaning.
Order the steps for estimating bias from training error.
Why use a very large training set when estimating bias?
Bias Is Measured on the Holdout Set
In the large-sample limit, bias is typically assessed on the _____ data.
Match each learning-term description with its role in the informal bias definition.
Order the steps for deciding whether bias is the main cause of poor model performance.
How Informal Bias Relates to Training Error
Interpreting Training Error on a Large Dataset
Informal Bias and Training Error