Learn Before
Explain why a voice-command model looks strong on one split but weak on another.
Case context: A voice-command recognition system achieves strong results on the training set and on a small validation set drawn from the same source as the training data. However, it performs much worse on the main development set.
Question: What problem best explains this pattern, and why does it happen?
Sample answer: The most likely issue is a data mismatch between the training data and the development data. The training set and the small validation set are similar, so the model does well there, but the main development set comes from a different distribution. Because the training data is not a good match for the development data, performance drops when the model is evaluated on the development set.
Key points:
- The correct diagnosis is data mismatch
- The model does well on the training set and the small validation set, but poorly on the development set
- The reason is that the training distribution does not match the development distribution
Rubric: The student must identify data mismatch and explain that the training data is a poor fit for the development data distribution.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Recognizing Distribution Mismatch from Error Measurements
Collect Training Examples That Resemble the Hard Development Cases
Compare training and development data properties after a mismatch is found
What best describes a mismatch between training data and evaluation data?
A data mismatch problem means the model performs poorly on both training-like data and the development/test data.
Data mismatch and distribution fit
Match each data-distribution term with its description.
Put the diagnostic workflow for dataset shift in the correct order.
What most directly causes a mismatch between training data and evaluation data?
A voice-to-text system can perform well on the training set and training-dev set but still do poorly on the dev set if the dev data comes from a different distribution.
Training Distribution and Evaluation Distribution
Match each performance pattern to the most likely diagnosis of the model's problem.
Put the steps in order for diagnosing a distribution mismatch from model performance.
Explain what a performance gap across two data distributions suggests
Explain why a voice-command model looks strong on one split but weak on another.
Why is this called data mismatch?