Google

Selective relabeling can be reasonable when you only need a practical result for building a product or application. It becomes a problem when the result will be reported in a research paper or when you need an unbiased estimate of test-set performance.

When Selective Relabeling Is Acceptable

When is it acceptable to clean up noisy labels even if the labels are not perfectly objective?

True or False: Correcting labels before reporting results in a research paper can create problematic label-correction bias.

Label-correction bias may be acceptable when the goal is simply to build a _____ or service.

Match each scenario to whether label correction is acceptable.

Question:
In a concise analytical response, explain why correcting uncertain or noisy labels may be acceptable in one context but not in another.

Sample answer:
If the goal is to improve a deployed system or guide product iteration, adjusting labels after reviewing mistakes can be reasonable because the main objective is better practical performance. However, if the result will be reported in a research paper or used as an unbiased estimate of model accuracy, the same adjustment becomes a problem because it can distort the measurement. In short, acceptability depends on whether the intended use can tolerate bias or requires a neutral evaluation.

Key points:
- Reasonable for product or system improvement
- Problematic for research reporting
- Problematic for an unbiased accuracy estimate
- Intended use determines whether the bias matters

Rubric:
A strong response states the development-use exception, identifies the research and unbiased-evaluation cases as problematic, and explains that acceptability depends on the goal.

Why Label Cleanup Can Be Acceptable in Some Settings

Case context:
A team is building a customer support classifier and first wants to use the result only for internal model debugging. Later, the same result may be included in a conference paper.

Question:
Determine when the relabeling bias is acceptable and when it becomes a problem.

Sample answer:
The bias is acceptable if the result is used only for internal debugging of the product. It becomes a problem once the team plans to present the result in a conference paper. It is also a problem if the team needs the result to serve as a fully unbiased estimate of test performance.

Key points:
- Internal debugging can tolerate the bias
- Conference-presentation use makes the bias problematic
- A requirement for an unbiased test-performance estimate also makes it problematic
- The judgment depends on how the result will be used

Rubric:
The response must clearly distinguish internal development use from conference-paper use and classify the bias appropriately in each case. Mentioning the need for an unbiased performance estimate strengthens the answer.

Decide when relabeling bias is acceptable in two planned uses of a result.

Question:
Answer in one to three sentences by naming the two situations in which correcting labels is not a good idea.

Sample answer:
Label correction is a poor choice if the corrected labels will be used in a scholarly publication or if you need an unbiased estimate of performance on the test set.

Key points:
- Use in a scholarly publication
- Need for an unbiased test-set performance estimate

Rubric:
Award full credit for identifying both the publication-use case and the need for an unbiased test-set estimate.

Learn Before

Related