Meaning of error on the training-like distribution
Question: In a credit-scoring example, the model achieves a 2.3% error rate on a holdout set drawn from the same distribution as the training data. What does this metric indicate about the model?
Sample answer: It indicates that the model generalizes well to that distribution, so severe overfitting is not the main problem on that data source.
Key points:
- The model performs well on the training-like distribution.
- High variance is not the primary issue on that distribution.
Rubric: The answer should state that the model does well on the training-like distribution or that overfitting/high variance is not the main issue there.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Which comparison most clearly indicates a data mismatch problem in the case where training error is 4%, same-distribution unseen error is 5%, and dev error is 14%?
A train error of 0.8%, a same-source holdout error of 1.0%, and a development error of 7.5% mainly show overfitting.
In the distribution-shift example, the classifier has _____ error on the dev set.
Match each error metric in the distribution-shift example to its value.
Order the steps for diagnosing a distribution-shift problem in model evaluation.
What does a 0.4% gap between training error (2.0%) and error on other data drawn from the same distribution (2.4%) suggest?
If training error is 2%, human-level error is 1%, and development error on a different data source is 12%, the 1% gap between training error and human-level error is the biggest issue to fix.
For unseen data drawn from the same distribution as the training set, the error is _____.
Match each comparison to the kind of error it measures.
Order the evidence showing that distribution shift is the main issue in a speech-command classifier.
Explain how two error comparisons reveal a distribution mismatch.
Diagnosing a Dev-Set Error Spike in a Product Review Classifier
Meaning of error on the training-like distribution