Why a Model Looks Strong in Validation but Weak on Deployment Data
Case context: A team trains a fraud-detection model and gets very strong results on its validation split. After deployment, however, the score drops sharply on live transactions. The validation data and the live data were collected from different sources and time periods.
Question: What are three plausible explanations for the drop in performance? Also, in which explanation would it be reasonable to think that no major additional gains are available from the current modeling approach?
Sample answer: Three plausible explanations are: 1) the model was tuned too specifically to the validation split, 2) the live data is intrinsically more difficult than the validation data, or 3) the live data comes from a meaningfully different distribution than the validation data. If the live data is simply harder, then the model may already be near the best performance that can be expected from the current approach, so large further improvements may not be realistic.
Key points:
- Tuning too specifically to the validation split
- Live data is harder
- Live data is different
- Little additional improvement may be possible if the live data is harder.
Rubric: The response must identify the three distinct reasons for the drop in performance and correctly state that the case where the live data is inherently harder is the one in which major further improvement may not be possible.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why a Mismatched Validation Set Can Waste Effort
Which explanation is NOT one of the standard reasons a model can score well on the dev set but poorly on the test set when the two sets come from different distributions?
True or False: If a development set and a test set come from different distributions, it is easy to pinpoint the reason a model scores worse on the test set.
If the development set and the test set come from different _____, a performance gap is hard to interpret.
Match each distribution-shift situation to the most likely explanation.
Order the three explanations for why a model can look strong on development data but weak on test data when the data distributions are different.
When the test data are drawn from a tougher distribution than the development data, what is the most reasonable conclusion?
True or False: If a model scores lower on the test set than on the dev set, then the test set must be inherently more difficult.
A model may do well on the _____ set and still perform poorly on the test set if the two sets come from different distributions.
Match each dev/test diagnosis to the most appropriate next step.
Arrange the steps a practitioner follows when a spam filter succeeds on validation data but fails after deployment.
Why a good development score can still leave test results unexplained
Why a Model Looks Strong in Validation but Weak on Deployment Data
Three Reasons a Held-Out Test Set Can Look Worse