How to investigate a dataset mismatch after a dev-set drop
Question: A classification model scores well on its training data but noticeably worse on the development set. What should you do first to start diagnosing the mismatch, and what should you compare between the two datasets?
Sample answer: When a model performs much better on the training set than on the development set, the first diagnostic step is to examine how the data differ across the two sets. In particular, compare the properties and distributions of the training and development examples to identify what has changed.
Key points:
- Recognize that a training-versus-development performance gap suggests a data mismatch.
- State the first recommended diagnostic action: examine the data differences.
- Focus on comparing the properties and distributions of the training set and the development set.
Rubric: The response must state that when a mismatch is detected because development-set performance is poor, the recommended first step is to investigate which data properties and distributions differ between the training and development sets.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Using Error Analysis to Investigate Dataset Mismatch
What Should You Do First When Training and Development Data Do Not Match?
True or false: If a model performs well on the training set but poorly on the dev set, a useful next step is to compare how the two datasets differ.
What should you compare when diagnosing a data mismatch?
When a training set and a validation set look different, what should you examine first?
When a model performs differently on two datasets from different populations, a useful first step is to identify which properties of the examples differ between the two sets.
When a train-dev mismatch is suspected, the key diagnostic is to compare which _____ differ across the two datasets.
Match each data-split term to its meaning in diagnosing distribution shift.
Put the steps for identifying a training-development mismatch in order.
Why compare training-set properties with dev-set properties when a model shows a mismatch problem?
True or False: A model that scores well on training data but much worse on dev data is probably underfitting.
A gap between _____ performance and dev-set performance can signal data mismatch.
Match each observation or action to its correct interpretation in a data mismatch diagnosis.
Order the reasoning for diagnosing a training-dev mismatch
How to investigate a dataset mismatch after a dev-set drop
Investigating a gap between training and validation performance in a restaurant review classifier
Comparing dataset characteristics after validation failure