Practical Convenience Causes Label-Correction Bias in Dev Sets
In a dev set of 1,000 examples with a classifier that has 98.0% accuracy, it is easier to examine the 20 misclassified examples than all 980 correctly classified examples. Because it is easier in practice to check only the misclassified examples, bias does creep into some dev sets.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Practical Convenience Causes Label-Correction Bias in Dev Sets
When Label-Correction Bias Is Acceptable Versus Problematic
What risk arises when you fix label errors only for the dev-set examples your classifier misclassified?
True or False: Correcting mislabeled dev-set examples only where your system was wrong produces an unbiased evaluation.
Fixing labels only on examples your system _____ can introduce bias into dev-set evaluation.
Why does fixing labels only on misclassified dev examples introduce bias into the evaluation?
Fixing labels only on misclassified dev examples can introduce bias into your evaluation.
To avoid label-correction bias, you should review labels of _____ dev examples, not only misclassified ones.
Match each label-correction practice to its effect on dev set evaluation bias.
Order the steps a team should follow when correcting dev set labels to avoid introducing bias.
What is the most likely effect on measured dev set accuracy when labels are corrected only on misclassified examples?
Reviewing only the dev examples your model misclassified is sufficient to ensure an unbiased dev set evaluation.
Label-correction bias arises because mislabeled examples the system classified _____ are never reviewed or fixed.
Match each term related to label-correction bias to its correct definition.
Order the reasoning steps that explain why fixing only misclassified labels introduces bias into dev set evaluation.
Explain how selective label correction on misclassified examples alters estimated dev set performance.
Evaluating a team's decision to correct only misclassified dev set labels.
State the primary risk of fixing only misclassified dev set labels.
Learn After
Why does label-correction bias tend to creep into dev sets during error analysis?
True or False: Checking only misclassified dev examples is easier because there are far fewer of them to review.
With 1,000 dev examples and 98.0% accuracy, it is easier to examine the _____ misclassified examples than all 980 correct ones.
Match each dev set quantity to its description for a 1,000-example set with 98.0% accuracy.
Order the reasoning that explains how practical convenience leads to label-correction bias.
Analyze why practical convenience, rather than a deliberate methodological choice, drives label-correction bias in dev sets.
Diagnose why a team's practice of only reviewing misclassified dev examples introduces bias.
In one to three sentences, explain why reviewing only misclassified dev examples is more practical than reviewing all examples.
What is the underlying consequence of only correcting labels on misclassified dev examples?
True or False: The 980 correctly classified examples are guaranteed to have accurate labels since they were not flagged for correction.