Deciding whether to build a manual review set for satellite burst detection.
Case context: A machine learning team is building a detector for rare burst noise in satellite telemetry. The bursts last only a few milliseconds and are so domain-specific that field technicians cannot reliably tell whether a flagged segment is truly a burst or a normal fluctuation. The team is considering whether to spend time creating an Eyeball dev set of flagged examples for human inspection.
Question: Given that humans cannot reliably judge these examples, what should the team do about the Eyeball dev set, and why?
Sample answer: They should skip the Eyeball dev set. If people cannot confidently tell whether a case is correct, then manual inspection will not help them understand why the model is wrong, so the set would not be a useful debugging tool.
Key points:
- Do not create the Eyeball dev set.
- Human reviewers cannot perform the task reliably.
- Without human judgment, error analysis will not explain model mistakes well.
Rubric: The answer should recommend skipping the Eyeball dev set and explain that manual review is not useful when humans cannot reliably do the underlying task or diagnose the model's errors.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why is a manual review set less useful when people cannot do the task reliably themselves?
An eyeball-style dev set is equally helpful even when humans cannot perform the task well.
When people are poor at a task, it is harder to explain why the model _____ a case the right way.
Match each situation to the correct guidance about whether an eyeball dev set is worthwhile.
Arrange the steps for deciding whether an Eyeball dev set is useful.
A company is training a system to detect tiny defects in industrial microscope images, but even expert inspectors cannot reliably tell when the system is wrong. What should they do about the Eyeball dev set?
A manually reviewed dev set may be omitted when people cannot reliably judge whether the model is correct.
If people cannot judge the task reliably, you might _____ an Eyeball dev set altogether.
Match the manual review concept to its role in model debugging.
Order the reasoning steps that explain why a manual review set adds little value when people cannot reliably do the task themselves.
When an Eyeball Dev Set Becomes Less Useful
Deciding whether to build a manual review set for satellite burst detection.
Why is manual review of mistakes less useful on tasks people cannot do reliably?