Learn Before
Evaluating a loan default model with weak training performance.
Case context: A team trains a model to predict whether a loan applicant will repay on time. After training, the team estimates the model’s bias at 15% and its variance at 1%. The model misses many examples from the training data.
Question: What does this tell you about the model, and what would you expect its dev-set error to look like compared with its training error?
Sample answer: The model is best described as underfitting: it has too much bias and very little variance. Since the variance is only 1%, performance on the dev set should be only a little worse than performance on the training set.
Key points:
- A 15% bias estimate together with 1% variance points to an overly simple model.
- Low variance means the dev error should stay close to the training error.
- The main problem is underfitting rather than overfitting.
Rubric: The response must identify the model as underfitting, or equivalent high-bias/low-variance behavior, and state that the dev-set error is only slightly above the training error.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
A spam filter has 18% training error and only 19% error on the dev set. What is the best description of this model?
When a model has high bias and low variance, the training error is usually much lower than the dev error because the model is overfitting.
A model that does not match the training data well and generalizes poorly is said to have high bias and low _____.
Match each term to its meaning in a simple train/dev error example.
Order the steps for estimating bias and variance in a classifier.
What does an estimated variance of 2% indicate when the estimated bias is 12%?
A model that underfits usually has low training error and much higher dev error.
Estimating Bias from Training Error
Link the fit pattern to what it says about a model
Arrange the reasoning steps that show a classifier with 18% training error and a 1% train-dev gap is underfitting.
Explain what 12% bias and 2% variance imply for a classifier.
Evaluating a loan default model with weak training performance.
What is the name for a model that is too simple to fit the patterns in the data well?