Comparing dataset characteristics after validation failure
Question: A spam filter does well on data drawn from the same source as its training set, but it performs poorly on a validation collection gathered from a different region. What should you examine first to diagnose the mismatch?
Sample answer: You should compare the training set and validation set to figure out which data characteristics are different between their distributions.
Key points:
- Compare the two datasets directly.
- Look for differences in data characteristics.
- Focus on how the distributions are not the same.
Rubric: The response must state that you should examine which characteristics differ between the training and validation data distributions.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Using Error Analysis to Investigate Dataset Mismatch
What Should You Do First When Training and Development Data Do Not Match?
True or false: If a model performs well on the training set but poorly on the dev set, a useful next step is to compare how the two datasets differ.
What should you compare when diagnosing a data mismatch?
When a training set and a validation set look different, what should you examine first?
When a model performs differently on two datasets from different populations, a useful first step is to identify which properties of the examples differ between the two sets.
When a train-dev mismatch is suspected, the key diagnostic is to compare which _____ differ across the two datasets.
Match each data-split term to its meaning in diagnosing distribution shift.
Put the steps for identifying a training-development mismatch in order.
Why compare training-set properties with dev-set properties when a model shows a mismatch problem?
True or False: A model that scores well on training data but much worse on dev data is probably underfitting.
A gap between _____ performance and dev-set performance can signal data mismatch.
Match each observation or action to its correct interpretation in a data mismatch diagnosis.
Order the reasoning for diagnosing a training-dev mismatch
How to investigate a dataset mismatch after a dev-set drop
Investigating a gap between training and validation performance in a restaurant review classifier
Comparing dataset characteristics after validation failure