Checking a Data Mismatch in Loading-Dock Speech
Case context: You are training a voice-command model for a warehouse headset. Most of the training audio is clean speech, but a small portion comes from a noisy loading dock. Your training dev set also includes a loading-dock subset. The model's overall accuracy is poor, and you suspect the main issue is that the model is not handling loading-dock speech well.
Question: Using the idea of checking a mismatch hypothesis on training and training dev subsets, what should you measure on these subsets, and what outcome would show that collecting more loading-dock recordings is the right next step?
Sample answer: Measure the model's performance separately on the loading-dock portion of the training set and on the loading-dock portion of the training dev set. If the model performs well on the training-set loading-dock examples but much worse on the training-dev loading-dock examples, that supports the idea that more recordings from the loading dock would likely improve the system.
Key points:
- Evaluate performance on the loading-dock subset in the training set.
- Evaluate performance on the loading-dock subset in the training dev set.
- A strong result on the training subset but a weak result on the training dev subset supports gathering more loading-dock data.
Rubric: The response must say to evaluate the loading-dock subset in both the training set and the training dev set, and must explain that good training-set performance combined with poor training-dev performance on that subset supports collecting more data from the loading-dock environment.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the most likely explanation if a model is accurate on the examples it was trained on but much less accurate on a separate development subset from the same source?
A speech-recognition system is trained on clean studio speech, and both the training set and the development set include a 'noisy train station' slice. What should you measure separately on that slice?
What Kind of Data Should Be Added After a Train–Dev Gap?
If both the training set and the training-dev set contain dashcam audio from city buses, what is the best first step for checking a data mismatch hypothesis?
A performance gap on the same minority slice supports the data-mismatch hypothesis.
A gap between training and the _____ usually signals overfitting.
Match each observation about a subway-audio subset to what it suggests about the distribution mismatch question.
Order the diagnostic steps for testing whether a mismatch between data sources is the problem.
What should you conclude if a speech model is accurate on studio-recorded clips in the training set but much less accurate on studio-recorded clips in the training dev set?
If a fraud detector performs well on mobile-app purchases that appear in both the training set and the training-dev set, but poorly on in-store purchases that appear only in the training-dev set, this evidence cannot make a data-mismatch hypothesis more plausible.
Check the Matching Data Subset
Match each term in the distribution-mismatch diagnostic process to its correct meaning.
Order the steps for checking whether more conference-room audio is likely to help a speech model.
Using subset comparisons to test whether more targeted data is needed
Checking a Data Mismatch in Loading-Dock Speech
Validating the Case for More Data from a Specific Subset