Why a good development score can still leave test results unexplained
Question: In a machine learning project, the development set and the final evaluation set come from different data sources. A model can look excellent during development but perform badly at the end. Explain the three explanations for that gap and why the team may not know what improvement to pursue.
Sample answer: If a model does very well on the development data but fails on the final evaluation data, there are three main possibilities. One possibility is that the model became too tuned to the development set. Another is that the final evaluation set is genuinely more difficult, so the lower score may not indicate a fixable mistake in the algorithm. A third is that the final data is not harder, only different, so the patterns learned for development no longer transfer well. Because these explanations point to different remedies, the team cannot tell whether to reduce overfitting, improve the model itself, or gather data that better matches the final setting.
Key points:
- The model may have tuned too closely to the development set.
- The final set may be more difficult.
- The final set may differ rather than be harder.
- The correct next step is uncertain.
Rubric: The essay should identify the three causes of the performance gap and explain that uncertainty about which cause is responsible makes improvement planning unclear.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why a Mismatched Validation Set Can Waste Effort
Which explanation is NOT one of the standard reasons a model can score well on the dev set but poorly on the test set when the two sets come from different distributions?
True or False: If a development set and a test set come from different distributions, it is easy to pinpoint the reason a model scores worse on the test set.
If the development set and the test set come from different _____, a performance gap is hard to interpret.
Match each distribution-shift situation to the most likely explanation.
Order the three explanations for why a model can look strong on development data but weak on test data when the data distributions are different.
When the test data are drawn from a tougher distribution than the development data, what is the most reasonable conclusion?
True or False: If a model scores lower on the test set than on the dev set, then the test set must be inherently more difficult.
A model may do well on the _____ set and still perform poorly on the test set if the two sets come from different distributions.
Match each dev/test diagnosis to the most appropriate next step.
Arrange the steps a practitioner follows when a spam filter succeeds on validation data but fails after deployment.
Why a good development score can still leave test results unexplained
Why a Model Looks Strong in Validation but Weak on Deployment Data
Three Reasons a Held-Out Test Set Can Look Worse