How are the two parts of a split dev set used?
Question: In one to three sentences, explain what the inspected portion and the untouched portion of the split dev set are for.
Sample answer: The inspected portion is used for manual error analysis so the team can look closely at mistakes, but it is easier to overfit to. The untouched portion is reserved for tuning decisions and for checking whether repeated inspection has caused the team to over-specialize on the examined slice.
Key points:
- Inspected portion: manual review and error analysis.
- Untouched portion: parameter or model tuning.
- The split provides a way to notice overfitting to the examined portion.
Rubric: The answer should clearly separate manual analysis from hands-off tuning and mention either the higher overfitting risk of the inspected portion or the diagnostic benefit of having a second portion.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Human-Review Dev Set
Blackbox Dev Set
Use the Full Dev Set When It Is Too Small to Split
Why split a development set into a review subset and a tuning subset?
Does examining part of a dev set more closely increase the risk of overfitting to it?
The hands-off portion of the dev set can still be used to tune _____.
Match each dev-set idea with its role or consequence.
Order the reasoning for managing a dev set that is too large to inspect fully by hand.
Explain how separating a reviewed subset from an untouched subset can reveal overfitting during error analysis.
How should a team organize a large validation set with many mistakes?
How are the two parts of a split dev set used?
Which result suggests a model has been tuned too closely to the hand-checked subset?
If a development set is too small to divide into separate analysis and tuning subsets, using the full set for both purposes is a reasonable choice.