Testing a Data-Mismatch Hypothesis on a Shared Subset
If both your training set and your training-dev set contain the same special subset of examples, evaluate that subset separately. For example, if both splits include calls recorded in a noisy subway station, compare performance on those clips in each split. If the model performs well on that subset in training but much worse on the same subset in training-dev, that supports the idea that getting more data from that source could improve the system.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Testing a Data-Mismatch Hypothesis on a Shared Subset
A wildlife classifier has 8,000 camera-trap photos from reserve staff and 160,000 unrelated web photos. How should the reserve photos be used?
Combining In-Domain and Auxiliary Examples During Training
Adding target-distribution examples to training means the training set now includes data from the _____ distribution.
Match the training-set components with their distribution role in a model built for a specific target domain.
Order the steps for allocating product-review data between target and auxiliary sources.
Why can it help to include some target-distribution audio in both the training set and the training-dev set?
True or False: It can still be worthwhile to train on a dataset that comes from a different distribution than the dev/test set.
Strong results on source examples but weak results on deployment examples validate the hypothesis that more _____ data would help.
Match each training choice for a voice-command system to its likely effect.
Order the steps for training a defect classifier with mixed data sources.
Why Include a Small Amount of Target-Distribution Data in Training?
Using Target-Distribution Samples to Guide Training Decisions
Checking Whether More Same-Distribution Data Would Help
Learn After
What is the most likely explanation if a model is accurate on the examples it was trained on but much less accurate on a separate development subset from the same source?
A speech-recognition system is trained on clean studio speech, and both the training set and the development set include a 'noisy train station' slice. What should you measure separately on that slice?
What Kind of Data Should Be Added After a Train–Dev Gap?
If both the training set and the training-dev set contain dashcam audio from city buses, what is the best first step for checking a data mismatch hypothesis?
A performance gap on the same minority slice supports the data-mismatch hypothesis.
A gap between training and the _____ usually signals overfitting.
Match each observation about a subway-audio subset to what it suggests about the distribution mismatch question.
Order the diagnostic steps for testing whether a mismatch between data sources is the problem.
What should you conclude if a speech model is accurate on studio-recorded clips in the training set but much less accurate on studio-recorded clips in the training dev set?
If a fraud detector performs well on mobile-app purchases that appear in both the training set and the training-dev set, but poorly on in-store purchases that appear only in the training-dev set, this evidence cannot make a data-mismatch hypothesis more plausible.
Check the Matching Data Subset
Match each term in the distribution-mismatch diagnostic process to its correct meaning.
Order the steps for checking whether more conference-room audio is likely to help a speech model.
Using subset comparisons to test whether more targeted data is needed
Checking a Data Mismatch in Loading-Dock Speech
Validating the Case for More Data from a Specific Subset