Learn Before
Why keep label corrections aligned across development and test data?
Question: In one to three sentences, explain the main reason for using the same label-correction procedure on both the development set and the test set.
Sample answer: The goal is to keep the development and test sets representative of the same underlying distribution. That way, the team does not tune the model to one evaluation standard on the development set and then face a different standard on the test set.
Key points:
- Keeps the development and test sets aligned with the same distribution
- Prevents tuning for one development criterion and being judged by a different test criterion
Rubric: The answer must mention both the shared distribution idea and the risk of optimizing for a development criterion that does not match the test criterion.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why should the same label-cleaning procedure be used for both the dev set and the test set?
Validation and Test Labels Must Be Handled Consistently
Any label-fixing rule you use for the development set should also be applied to the _____ labels.
Match each relabeling situation to its effect on dev and test evaluation.
Order the steps for correcting label mistakes while keeping evaluation sets aligned.
What is the main problem if the dev set and test set are labeled using different rules?
It is fine to use one procedure to clean labels for the validation set and a different procedure for the test set if both look accurate overall.
Aligning validation and test label fixes helps prevent the team from optimizing for one score and then being judged by a _____ scoring rule.
Match each label-cleaning concept to its definition.
Order the steps that create misleading evaluation when only validation labels are corrected.
Explain what happens if a team cleans labels only in the validation set and leaves the test set unchanged.
Explain the evaluation mismatch when only one dataset receives label cleanup.
Why keep label corrections aligned across development and test data?