Why split the validation set into reviewed and unreviewed parts?
Question: Explain why separating the validation set into a manually reviewed slice and an untouched slice helps assess the effect of human error analysis.
Sample answer: The split lets you compare results on the reviewed slice with results on the untouched slice. If performance improves only on the reviewed slice, that is a sign that the manual analysis process is overfitting that slice instead of improving general validation performance.
Key points:
- Compares the manually reviewed slice against the untouched slice.
- Helps detect overfitting caused by the manual analysis process.
- Shows whether gains are real or limited to the reviewed subset.
Rubric: The answer should say that the split makes it possible to detect overfitting to the reviewed subset by comparing it with the untouched subset.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Overfitting a development set used for model tuning
What most clearly indicates that a manually inspected dev slice has been over-tuned during error analysis?
Inspecting a manually reviewed validation set can make you adapt to that set more quickly than if you never looked at its examples.
Keeping a Reviewed Subset Separate from a Hidden Check Set
Match each validation-set concept to its description in a reviewed-vs-reserved split.
Order the steps for checking whether repeated manual review is causing overfitting to a reviewed validation subset.
What is the most likely conclusion when manual review keeps pushing one dev set score far above another?
In the Eyeball/Blackbox approach, the Blackbox dev set is checked manually during routine error analysis.
If the Eyeball dev set improves much faster than the Blackbox dev set, you have _____ the Eyeball dev set.
Match each performance pattern with the most appropriate interpretation when comparing a hand-reviewed set with a separate hidden set.
Order the steps for checking whether a manually reviewed dev set has been overfit.
Detect Overfitting to an Inspected Development Set
When Manual Review Improves Faster Than a Hidden Evaluation Set
Why split the validation set into reviewed and unreviewed parts?