Learn Before
Why mislabeled development examples matter more as models improve
When a model is still making many mistakes, a few incorrect labels in the development or test set may have little effect on the overall error picture. As the model gets better and its true error rate drops, those same mislabeled examples can account for a larger share of the observed errors. At that point, improving the label quality of the development set can become worth the effort.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why mislabeled development examples matter more as models improve
When should mislabeled development-set examples be fixed?
When to Repair Validation Labels
Noisy labels can ____ the comparison itself.
Match each development-set concept to its role in deciding whether labels should be fixed.
Order the reasoning for deciding whether a noisy development set should be corrected.
When should mislabeled dev-set examples be fixed?
Decide When Incorrect Validation Labels Should Be Fixed
When is it worth fixing mislabeled validation examples?
When is cleaning dev-set labels worth the effort?
Fixing validation labels one by one is always required.
Learn After
Why do labeling mistakes in a validation set matter more after a classifier gets stronger?
A small number of mislabeled validation or test examples may be acceptable at first, and that decision can be revisited later.
When Label Cleanup Becomes Worth the Effort
Match each development-set scenario with its implication for mislabeled examples.
Order the steps for deciding whether label cleanup on a development set is worth the effort.
A speech-recognition dev set has about 3% error, and 40% of those errors come from incorrectly transcribed examples. What should you do?
A model error rate of 1.4% versus 2.0% is a small difference that usually does not matter.
How mislabeled dev examples matter after the model gets better
Match each concept to its role when mislabeled development examples become more important.
Order the stages showing why mislabeled development examples matter more as a classifier improves.
When and why mislabeled development examples become more costly to ignore
Decide whether to relabel the dev set once label noise becomes a large share of remaining errors.
Why does noisy dev-set labeling matter more as a model improves?