Learn Before
Explain what a performance gap across two data distributions suggests
Question: A classifier performs well on new examples that resemble its training data, but its accuracy drops on a different evaluation set collected from another source. Explain why this can happen and what the gap says about the training data.
Sample answer: This pattern points to a data mismatch problem. The model does well on new examples from the training distribution because it has learned that pattern of data. It performs worse on the evaluation set from the other source because those examples come from a different distribution. The gap suggests that the training set is not a good match for the evaluation data.
Key points:
- The model generalizes well to new data drawn from the training distribution.
- The model performs worse on data drawn from a different dev/test distribution.
- This difference in performance is a sign of data mismatch.
- The training set does not adequately reflect the dev/test distribution.
Rubric: The answer should state that the model can succeed on data similar to its training examples but fail on the dev/test set because the two distributions differ, and that this indicates the training data is a poor match for the evaluation data.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Recognizing Distribution Mismatch from Error Measurements
Collect Training Examples That Resemble the Hard Development Cases
Compare training and development data properties after a mismatch is found
What best describes a mismatch between training data and evaluation data?
A data mismatch problem means the model performs poorly on both training-like data and the development/test data.
Data mismatch and distribution fit
Match each data-distribution term with its description.
Put the diagnostic workflow for dataset shift in the correct order.
What most directly causes a mismatch between training data and evaluation data?
A voice-to-text system can perform well on the training set and training-dev set but still do poorly on the dev set if the dev data comes from a different distribution.
Training Distribution and Evaluation Distribution
Match each performance pattern to the most likely diagnosis of the model's problem.
Put the steps in order for diagnosing a distribution mismatch from model performance.
Explain what a performance gap across two data distributions suggests
Explain why a voice-command model looks strong on one split but weak on another.
Why is this called data mismatch?