Fix a training-dev distribution gap
Case context: You are building a defect detector for used battery packs. The model achieves strong results on examples collected in a controlled lab, where the batteries are clean and evenly lit. But when you evaluate it on the development set, most items come from a busy warehouse line with dust, shadows, and inconsistent camera angles, and the error rate is much higher.
Question: What problem is the system experiencing, and what focused action should you take to help the algorithm handle the hard development examples?
Sample answer: This is a data distribution mismatch: the examples used to train the model are not similar to the examples it struggles with during development. The best next step is to collect additional labeled battery-pack images from the warehouse setting and include them in the training set so the model sees more of the same kind of data.
Key points:
- Identify the issue as a mismatch between the training set and the development set.
- Recognize that the lab images and warehouse images come from different distributions.
- Recommend adding more training examples that resemble the difficult development cases.
- State that the added examples should come from the warehouse environment.
Rubric: Responses must correctly identify the issue as a data distribution mismatch and recommend the specific action of collecting more warehouse-style training images.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Data Mismatch May Not Have a Clear Fix
Synthetic Data That Approximates the Dev Distribution
Are Synthetic Training Examples Representative?
What should you do when training results are strong but validation results drop because the data sources differ?
A data mismatch problem happens when a model does well on training data but performs poorly on a dev set that comes from a different distribution.
Matching the Dev Set Environment
Match each data-mismatch idea to its description.
Order the steps for diagnosing and fixing a data mismatch in a customer-feedback classifier.
What most likely explains the model’s weak performance on the development set in this speech project?
Does training on examples that look more like the dev set always eliminate data mismatch?
When a speech recognizer performs poorly on noisy clips in the dev set, one remedy is to collect more training data that better _____ those difficult examples.
Match each part of a traffic-sign recognition scenario to its role in a data mismatch diagnosis.
Order the steps for deciding whether to collect training data that better matches difficult dev examples.
Using Targeted Data to Reduce Distribution Mismatch
Fix a training-dev distribution gap
What training data change helps with a data mismatch problem?