Learn Before
Decide when relabeling bias is acceptable in two planned uses of a result.
Case context: A team is building a customer support classifier and first wants to use the result only for internal model debugging. Later, the same result may be included in a conference paper.
Question: Determine when the relabeling bias is acceptable and when it becomes a problem.
Sample answer: The bias is acceptable if the result is used only for internal debugging of the product. It becomes a problem once the team plans to present the result in a conference paper. It is also a problem if the team needs the result to serve as a fully unbiased estimate of test performance.
Key points:
- Internal debugging can tolerate the bias
- Conference-presentation use makes the bias problematic
- A requirement for an unbiased test-performance estimate also makes it problematic
- The judgment depends on how the result will be used
Rubric: The response must clearly distinguish internal development use from conference-paper use and classify the bias appropriately in each case. Mentioning the need for an unbiased performance estimate strengthens the answer.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
When is it acceptable to clean up noisy labels even if the labels are not perfectly objective?
True or False: Correcting labels before reporting results in a research paper can create problematic label-correction bias.
Label-correction bias may be acceptable when the goal is simply to build a _____ or service.
Match each scenario to whether label correction is acceptable.
Order the steps for deciding whether label-based correction is acceptable
Why Label Cleanup Can Be Acceptable in Some Settings
When is label correction a poor choice?
A project needs an unbiased held-out accuracy estimate. How should it view relabeling errors in that set?
A small product team and a publishable research study should always handle label fixes in the same way.
Related
When is it acceptable to clean up noisy labels even if the labels are not perfectly objective?
True or False: Correcting labels before reporting results in a research paper can create problematic label-correction bias.
Label-correction bias may be acceptable when the goal is simply to build a _____ or service.
Match each scenario to whether label correction is acceptable.
Order the steps for deciding whether label-based correction is acceptable
Why Label Cleanup Can Be Acceptable in Some Settings
When is label correction a poor choice?
A project needs an unbiased held-out accuracy estimate. How should it view relabeling errors in that set?
A small product team and a publishable research study should always handle label fixes in the same way.