Why mismatched evaluation data can mislead a project
Question: Discuss why a model team can be misled if its development and test data come from a different source than the data the model will face in production, and state what they should do.
Sample answer: If the holdout data comes from a distribution unlike the production data, the score on that holdout set may not predict real-world behavior. The team may optimize for the wrong pattern and believe the system is improving when deployment performance is not. The fix is to rebuild or revise the development and test sets so they better match the target distribution the model must handle.
Key points:
- Holdout data should reflect the data the system will encounter.
- A mismatch makes evaluation results unreliable for deployment decisions.
- The correct response is to update the development and test sets to match the target distribution more closely.
Rubric: The response should explain that a mismatch between holdout and production data gives misleading evaluation signals because the holdout is not representative, and it should state that the development and test sets need to be revised to better match the target distribution.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
When the Development Set No Longer Matches Real User Data
If a dev/test set does not reflect the distribution the model will face after deployment, what should the team do?
True or False: A validation set drawn from a very different population than deployment data can still reliably show which model changes will help in the real world.
When the validation set does not match deployment conditions, the recommended action is to _____ the validation set.
Identify the distribution concepts used in model evaluation.
Arrange the actions a team should take when a validation set stops predicting real-world performance.
Why a Non-Representative Dev/Test Set Can Mislead Model Improvement
True or False: Evaluation data should match the conditions the final system will face in practice.
Core idea about dev and test data
Match each situation to the best action for dev/test data.
Order the reasoning steps showing why a development set that misses the real distribution should be revised.
Why mismatched evaluation data can mislead a project
Diagnose a field-to-test mismatch in apple inspection photos
What should a team do if its evaluation data no longer reflects real-world use?