Learn Before
Recognizing Distribution Mismatch from Error Measurements
A distribution mismatch is likely when the model does very well on the training set, only slightly worse on other examples drawn from the same source, but performs much worse on a dev set that comes from a different source or setting. For example, a system with 0.9% training error, 1.4% error on a same-source holdout set, and 9.8% dev error shows a large gap that points to data mismatch rather than simple overfitting alone.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Recognizing Distribution Mismatch from Error Measurements
Collect Training Examples That Resemble the Hard Development Cases
Compare training and development data properties after a mismatch is found
What best describes a mismatch between training data and evaluation data?
A data mismatch problem means the model performs poorly on both training-like data and the development/test data.
Data mismatch and distribution fit
Match each data-distribution term with its description.
Put the diagnostic workflow for dataset shift in the correct order.
What most directly causes a mismatch between training data and evaluation data?
A voice-to-text system can perform well on the training set and training-dev set but still do poorly on the dev set if the dev data comes from a different distribution.
Training Distribution and Evaluation Distribution
Match each performance pattern to the most likely diagnosis of the model's problem.
Put the steps in order for diagnosing a distribution mismatch from model performance.
Explain what a performance gap across two data distributions suggests
Explain why a voice-command model looks strong on one split but weak on another.
Why is this called data mismatch?
Learn After
Which comparison most clearly indicates a data mismatch problem in the case where training error is 4%, same-distribution unseen error is 5%, and dev error is 14%?
A train error of 0.8%, a same-source holdout error of 1.0%, and a development error of 7.5% mainly show overfitting.
In the distribution-shift example, the classifier has _____ error on the dev set.
Match each error metric in the distribution-shift example to its value.
Order the steps for diagnosing a distribution-shift problem in model evaluation.
What does a 0.4% gap between training error (2.0%) and error on other data drawn from the same distribution (2.4%) suggest?
If training error is 2%, human-level error is 1%, and development error on a different data source is 12%, the 1% gap between training error and human-level error is the biggest issue to fix.
For unseen data drawn from the same distribution as the training set, the error is _____.
Match each comparison to the kind of error it measures.
Order the evidence showing that distribution shift is the main issue in a speech-command classifier.
Explain how two error comparisons reveal a distribution mismatch.
Diagnosing a Dev-Set Error Spike in a Product Review Classifier
Meaning of error on the training-like distribution