Learn Before
Manual Review Sets Can Be Unhelpful for Tasks People Cannot Judge Reliably
When people cannot perform a task with enough accuracy to spot model mistakes, a manually reviewed dev set often adds little value. Human reviewers may struggle to tell whether a prediction is truly wrong or to understand what caused the error, so it can be reasonable to leave out an Eyeball-style dev set and use other evaluation methods instead.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Spotting Overfitting to a Manually Reviewed Dev Slice
How to Size an Eyeball Dev Set When Data Is Easy to Review
A Review Set Should Expose the Main Error Types
Manual Review Sets Can Be Unhelpful for Tasks People Cannot Judge Reliably
What is the main purpose of the human-review subset when a validation set is split into two parts?
The label "manual review set" can remind a team that people should inspect those examples directly.
What fraction of the dev set is manually reviewed in the Eyeball subset?
Match each dev set split term to its description.
Arrange the steps for building and using a small review subset from a larger development set in the correct order.
If 20% of a 3,000-example dev set is set aside as the Eyeball dev set, how many examples are in that subset?
If a 400-example Eyeball dev set has the same 20% error rate as the full dev set, you would expect about 80 misclassified examples.
The eye-check dev set should contain enough mistakes for you to _____.
Match each manual-review subset fact to the idea it describes.
Order the steps for deciding whether a small inspection subset is large enough for error analysis.
How should the manual-review subset be named in a voice transcription project?
Split the Development Set Into an Inspection Set and a Locked Set
How large should an Eyeball dev set be for useful manual error review?
Learn After
Why is a manual review set less useful when people cannot do the task reliably themselves?
An eyeball-style dev set is equally helpful even when humans cannot perform the task well.
When people are poor at a task, it is harder to explain why the model _____ a case the right way.
Match each situation to the correct guidance about whether an eyeball dev set is worthwhile.
Arrange the steps for deciding whether an Eyeball dev set is useful.
A company is training a system to detect tiny defects in industrial microscope images, but even expert inspectors cannot reliably tell when the system is wrong. What should they do about the Eyeball dev set?
A manually reviewed dev set may be omitted when people cannot reliably judge whether the model is correct.
If people cannot judge the task reliably, you might _____ an Eyeball dev set altogether.
Match the manual review concept to its role in model debugging.
Order the reasoning steps that explain why a manual review set adds little value when people cannot reliably do the task themselves.
When an Eyeball Dev Set Becomes Less Useful
Deciding whether to build a manual review set for satellite burst detection.
Why is manual review of mistakes less useful on tasks people cannot do reliably?