Diagnosing a model with 14% training error, 15% train-dev error, and 16% dev error
Case context: A team is building a fraud detection model for online payments. Its measured error rates are 14% on the training set, 15% on the training-dev set, and 16% on the dev set.
Question: What is the most likely problem indicated by these numbers, and what do they suggest about the model's performance on the training data?
Sample answer: The most likely issue is high avoidable bias. The model is already making many mistakes on the training set, so it is not fitting the training data well and is underperforming there.
Key points:
- Identify high avoidable bias as the main problem.
- Recognize that training error is already substantial.
- Conclude that the model is not learning the training examples effectively.
Rubric: The response must identify the problem as high avoidable bias and clearly state that the model is performing poorly on the training data.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What does a pattern of 10% training error, 11% training-dev error, and 12% dev error most strongly suggest?
High training error signals poor fit on the training set
What type of bias is suggested when training error stays high?
Match each error metric in this classifier-debugging scenario to its correct value or diagnosis.
Diagnose High Bias from Three Error Rates
What does a tiny gap between training error and training-dev error indicate?
A model with 10% error on training data, 11% error on a training-like holdout set, and 12% error on the development set is mainly overfitting.
If training error is far worse than training-dev error, the model is doing poorly on the _____ set.
Diagnosing Error Sources from Training and Development Gaps
Diagnosing a High-Bias Model from Three Similar Error Rates
What a 10% train error, 11% train-dev error, and 12% dev error reveal
Diagnosing a model with 14% training error, 15% train-dev error, and 16% dev error
Recognizing the Bias Pattern in the Error Rates