When Evaluation Data Does Not Match Deployment Data
If the examples in the dev or test set come from a different distribution than the data the system must handle in production, the evaluation will not give a reliable signal for improvement. In that situation, the dev and test sets should be revised so they better represent the real operating environment.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
When Evaluation Data Does Not Match Deployment Data
When Repeated Validation Checks Distort Model Selection
When the Metric Rewards the Wrong Goal
When should your validation setup be revised?
True or False: If your initial validation split or evaluation metric turns out to be poorly chosen, you cannot revise it without abandoning the project.
If your evaluation metric no longer reflects your main objective, what should you change?
What is the clearest sign that your dev/test set or evaluation metric may need revision?
If a validation set or metric turns out to be poorly matched to the real goal, the team should rebuild the whole project before making any changes.
What to revise when the evaluation no longer matches the goal
Match each reason a validation metric can mislead the team to the recommended remedy.
What should a team do when its evaluation setup stops matching its goal?
When Validation Data Does Not Match Deployment Data
After revising your dev/test sets or evaluation metric, updating the project documentation is enough; the team does not need to be told about the new direction.
What should be expanded after repeated tuning to the validation set?
Match each situation to the underlying problem category it illustrates.
Order the reasoning steps for deciding whether to replace an evaluation metric that no longer matches the product goal.
When validation results stop matching the best product choice
When Evaluation Scores and Product Needs Disagree
What should a team do after the development set stops guiding decisions?
Learn After
When the Development Set No Longer Matches Real User Data
If a dev/test set does not reflect the distribution the model will face after deployment, what should the team do?
True or False: A validation set drawn from a very different population than deployment data can still reliably show which model changes will help in the real world.
When the validation set does not match deployment conditions, the recommended action is to _____ the validation set.
Identify the distribution concepts used in model evaluation.
Arrange the actions a team should take when a validation set stops predicting real-world performance.
Why a Non-Representative Dev/Test Set Can Mislead Model Improvement
True or False: Evaluation data should match the conditions the final system will face in practice.
Core idea about dev and test data
Match each situation to the best action for dev/test data.
Order the reasoning steps showing why a development set that misses the real distribution should be revised.
Why mismatched evaluation data can mislead a project
Diagnose a field-to-test mismatch in apple inspection photos
What should a team do if its evaluation data no longer reflects real-world use?