Why Different Dev and Test Distributions Complicate Diagnosis
When development and test sets come from different distributions, strong dev performance followed by weak test performance has several possible causes. The model may have overfit the dev set, the test distribution may contain intrinsically harder examples, or the model may generalize poorly across the distribution shift. Because these causes imply different remedies, the score gap alone does not identify what to fix.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Make the Dev Set Match the Main Improvement Goal
Mismatched Validation and Test Splits Can Make Chance Matter More
If a model is tuned on a development set and then performs worse on a separate test set, even though both sets come from the same source distribution, what is the most likely explanation?
True or False: If validation and test data are drawn from different populations, a score gap between them always has one clear cause.
If a model has started fitting the validation set too closely and the training and validation data come from the same distribution, the usual remedy is to get more _____ data.
Why should the development set match the main goal of the project?
When validation and test data come from the same distribution, a strong validation score followed by a much weaker test score suggests overfitting to the validation set.
If the development set is being overused and the training and development data come from the same distribution, the practical fix is to collect more _____ data.
Match each development-and-test-set situation with its consequence for debugging.
Order the troubleshooting steps when validation performance is strong but holdout performance is weak.
Why a Test Set Can Score Lower Than a Dev Set
A test score from a different data distribution tells you exactly why the model failed.
After the development and test sets are set, the team will spend its effort improving _____ set performance.
Match each concept about dev and test set distributions to the correct description.
How to Choose Dev and Test Sets for Reliable Iteration
Explain what poor final performance means when the tuning set and final set do or do not match.
Why a model can look strong in development but weak in final evaluation
What is the diagnosis and remedy when test results lag far behind dev results under the same data distribution?
A Same-Distribution Dev/Test Gap Suggests Dev-Set Overfitting
Why Different Dev and Test Distributions Complicate Diagnosis
Learn After
Which explanation is NOT one of the standard reasons a model can score well on the dev set but poorly on the test set when the two sets come from different distributions?
True or False: If a development set and a test set come from different distributions, it is easy to pinpoint the reason a model scores worse on the test set.
If the development set and the test set come from different _____, a performance gap is hard to interpret.
Match each distribution-shift situation to the most likely explanation.
Order the three explanations for why a model can look strong on development data but weak on test data when the data distributions are different.
When the test data are drawn from a tougher distribution than the development data, what is the most reasonable conclusion?
True or False: If a model scores lower on the test set than on the dev set, then the test set must be inherently more difficult.
A model may do well on the _____ set and still perform poorly on the test set if the two sets come from different distributions.
Match each dev/test diagnosis to the most appropriate next step.
Arrange the steps a practitioner follows when a spam filter succeeds on validation data but fails after deployment.
Why a good development score can still leave test results unexplained
Why a Model Looks Strong in Validation but Weak on Deployment Data
Three Reasons a Held-Out Test Set Can Look Worse
Why a Mismatched Validation Set Can Waste Effort