Learn Before
Reviewing Only Mistakes Can Skew a Dev Set
In a development set with 1,000 examples, a classifier with 98.0% accuracy makes about 20 mistakes and gets about 980 examples right. Because it is much faster to inspect the 20 mistakes than to audit all 980 correct predictions, teams often revise only the mislabeled cases. That convenience can introduce label bias into the development set.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Reviewing Only Mistakes Can Skew a Dev Set
When Selective Relabeling Is Acceptable
What problem can occur if you correct only the dev-set examples your model got wrong?
True or False: If you relabel only the validation examples your system missed, the validation result remains an unbiased estimate.
Fixing labels only on the development examples your model misclassifies can introduce bias into evaluation because those examples are not selected at random.
How can selective relabeling skew a development-set score?
Editing Only the Wrongly Labeled Dev Examples Can Skew Your Evaluation
To avoid label-fix bias, review labels on _____ dev examples, not only the ones your model got wrong.
Match each review policy to its impact on validation-set fairness.
Put the label-review workflow in the right order to avoid bias.
What happens to the measured development accuracy if labels are corrected only for examples the model got wrong?
Checking only the dev examples your model got wrong is enough to guarantee an unbiased evaluation set.
Selective relabeling misses examples the classifier gets _____, so some bad labels are never examined.
Match each label-audit term with its meaning.
Put the steps in order to show why correcting only mislabeled mistakes can distort validation results.
Why does correcting only the labels of examples the model got wrong distort measured accuracy?
Why reviewing only mistaken labels can distort validation results
State the main risk of fixing only misclassified dev set labels.
Learn After
What is the main reason error-checking can become biased when revising labels in a dev set?
Reviewing the Mistakes
Counting Misclassifications in a Development Set
Match Each Validation-Set Quantity to Its Meaning
Why fixing only the mistakes can distort a development set
Explain why label checks often get focused on the examples a model gets wrong.
Explain the bias created when a team inspects only the validation errors.
Why Focus on the Incorrect Dev Predictions?
What problem can arise when you review only the validation examples the system gets wrong?
True or False: The 1,120 examples that were classified correctly must all have perfectly accurate labels because they were not selected for review.