Why does noisy dev-set labeling matter more as a model improves?
Question: In a machine learning project, why can label mistakes in the dev set become a bigger problem after the model gets much better?
Sample answer: As model performance improves, the number of genuine mistakes in the dev set goes down. If the label noise stays the same, a larger share of the remaining measured errors comes from bad labels rather than from the model itself. That makes the evaluation less trustworthy, so cleaning up the dev-set labels becomes more valuable.
Key points:
- Model errors decrease as performance improves.
- Fixed label noise takes up a larger share of the remaining errors.
- This weakens the reliability of the evaluation metric.
- Improving label quality becomes worthwhile.
Rubric: The answer must explain that, as true model errors shrink, mislabeled dev examples account for a larger portion of the remaining error and make evaluation less reliable.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why do labeling mistakes in a validation set matter more after a classifier gets stronger?
A small number of mislabeled validation or test examples may be acceptable at first, and that decision can be revisited later.
When Label Cleanup Becomes Worth the Effort
Match each development-set scenario with its implication for mislabeled examples.
Order the steps for deciding whether label cleanup on a development set is worth the effort.
A speech-recognition dev set has about 3% error, and 40% of those errors come from incorrectly transcribed examples. What should you do?
A model error rate of 1.4% versus 2.0% is a small difference that usually does not matter.
How mislabeled dev examples matter after the model gets better
Match each concept to its role when mislabeled development examples become more important.
Order the stages showing why mislabeled development examples matter more as a classifier improves.
When and why mislabeled development examples become more costly to ignore
Decide whether to relabel the dev set once label noise becomes a large share of remaining errors.
Why does noisy dev-set labeling matter more as a model improves?