Diagnosing a Training–Validation Gap
Case context: You are building a spam classifier. It performs well on the training emails, but its accuracy drops sharply on a held-out validation set, suggesting that the training data and validation data may not match well.
Question: According to basic error-analysis practice for this kind of mismatch, what should you compare to identify the underlying cause?
Sample answer: You should compare the training set and the validation set so you can identify how they differ and determine whether those differences explain the performance drop.
Key points:
- Compare the training set with the validation set.
- Look for meaningful differences between the two sets.
- Use those differences to explain the mismatch.
Rubric: The answer must say that the analysis should examine differences between the training set and the validation set.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Train-Station Audio as a Speech Recognition Data Mismatch Example
What is the main purpose of error analysis when a model works well on suburban data but fails more often on downtown data?
Data Mismatch Is Diagnosed on the Test Set
The goal of error analysis in a data mismatch review is to identify the important _____ between the training data and the dev data.
Match each item to its role in a data mismatch review.
Arrange the steps for diagnosing a mismatch between training performance and dev-set performance.
What most directly causes a data mismatch between training and development performance?
Goal of Error Analysis in a Data-Mismatch Study
In a dataset-mismatch review, compare the training set against the _____ set.
Match each step in a distribution-gap review to its main purpose.
Order the steps for diagnosing a training–validation mismatch using error analysis.
Using error analysis to investigate a train-dev mismatch
Diagnosing a Training–Validation Gap
Purpose of error analysis in a mismatch study