Using subset comparisons to test whether more targeted data is needed
Question: Suppose you have identified a special subset of examples that might be causing a mismatch, and you mark that subset inside both the training set and the training-dev set. Explain how comparing the model's results on those two subsets can tell you whether collecting more examples of that type is a good idea.
Sample answer: First, measure the model on that same subset in both places. If the model does well on the subset within the training set but performs noticeably worse on that subset in the training-dev set, it suggests the model is not generalizing well to that kind of example. That pattern supports the idea that gathering additional data of that specific type would likely improve performance.
Key points:
- Isolate the same subset in both the training set and the training-dev set
- Compare the model's behavior on that subset across the two splits
- If the training subset score is much better than the training-dev subset score, the mismatch hypothesis is strengthened and more data of that type may help
Rubric: The response must explain that the same subset should be evaluated in both the training and training-dev sets, and that strong training-subset performance combined with weak training-dev subset performance supports collecting more of that subset data.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the most likely explanation if a model is accurate on the examples it was trained on but much less accurate on a separate development subset from the same source?
A speech-recognition system is trained on clean studio speech, and both the training set and the development set include a 'noisy train station' slice. What should you measure separately on that slice?
What Kind of Data Should Be Added After a Train–Dev Gap?
If both the training set and the training-dev set contain dashcam audio from city buses, what is the best first step for checking a data mismatch hypothesis?
A performance gap on the same minority slice supports the data-mismatch hypothesis.
A gap between training and the _____ usually signals overfitting.
Match each observation about a subway-audio subset to what it suggests about the distribution mismatch question.
Order the diagnostic steps for testing whether a mismatch between data sources is the problem.
What should you conclude if a speech model is accurate on studio-recorded clips in the training set but much less accurate on studio-recorded clips in the training dev set?
If a fraud detector performs well on mobile-app purchases that appear in both the training set and the training-dev set, but poorly on in-store purchases that appear only in the training-dev set, this evidence cannot make a data-mismatch hypothesis more plausible.
Check the Matching Data Subset
Match each term in the distribution-mismatch diagnostic process to its correct meaning.
Order the steps for checking whether more conference-room audio is likely to help a speech model.
Using subset comparisons to test whether more targeted data is needed
Checking a Data Mismatch in Loading-Dock Speech
Validating the Case for More Data from a Specific Subset