Investigating a gap between training and validation performance in a restaurant review classifier
Case context: You build a sentiment model for restaurant reviews, and it performs very well on the examples used for training. When you test it on a separate development set gathered from another app, the accuracy drops noticeably. This suggests the two datasets may not match well.
Question: What is the first analysis you should perform, based on the recommended troubleshooting approach, to understand this performance gap?
Sample answer: Start by comparing the training data and the development data to find out which characteristics are different. Check for changes in source, wording, label balance, noise, or other data features that could explain the drop.
Key points:
- A large drop on the development set can signal that the two datasets come from different distributions.
- The next step is to inspect and compare the data itself instead of adjusting the model blindly.
- The goal is to identify the specific ways the two datasets do not line up.
Rubric: The learner must say that the next recommended step is to determine which properties differ between the training data and the development data.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Using Error Analysis to Investigate Dataset Mismatch
What Should You Do First When Training and Development Data Do Not Match?
True or false: If a model performs well on the training set but poorly on the dev set, a useful next step is to compare how the two datasets differ.
What should you compare when diagnosing a data mismatch?
When a training set and a validation set look different, what should you examine first?
When a model performs differently on two datasets from different populations, a useful first step is to identify which properties of the examples differ between the two sets.
When a train-dev mismatch is suspected, the key diagnostic is to compare which _____ differ across the two datasets.
Match each data-split term to its meaning in diagnosing distribution shift.
Put the steps for identifying a training-development mismatch in order.
Why compare training-set properties with dev-set properties when a model shows a mismatch problem?
True or False: A model that scores well on training data but much worse on dev data is probably underfitting.
A gap between _____ performance and dev-set performance can signal data mismatch.
Match each observation or action to its correct interpretation in a data mismatch diagnosis.
Order the reasoning for diagnosing a training-dev mismatch
How to investigate a dataset mismatch after a dev-set drop
Investigating a gap between training and validation performance in a restaurant review classifier
Comparing dataset characteristics after validation failure