How to handle a validation set that is too small to split
Scenario: A research group has a small validation set and wants to keep a separate slice for quick human inspection, but there are not enough examples to carve out another subset. They still need to review mistakes, choose among candidate models, and adjust training settings.
Question: Which data set should serve as the team’s inspection set, and what should it be used for?
Sample answer: Use the full validation set as the inspection set. Apply it to error review, model comparison, and hyperparameter adjustment.
Key points:
- If the validation set is too small, keep it intact instead of splitting it further.
- Treat the complete validation set as the inspection set.
- Use it to study examples the model gets wrong.
- Use it to choose between models and tune hyperparameters.
Rubric: Full credit requires selecting the entire validation set, identifying its inspection role, and including all three uses. Partial credit applies when the set choice is correct but one or more uses are missing.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the best use of a development set that is too small to split?
A very small development set can be kept as one set instead of being split.
If the development set cannot be divided, use the _____ development set as the eyeball set.
Match each item to its role in a small validation-set workflow.
Put the steps for deciding what to do with a very small development set in order
What to do with a too-small development set
How to handle a validation set that is too small to split
Three functions of a very small validation set
Best use of a dev set that is too small to split
If the full development set is used as an eyeball set, it cannot be used for hyperparameter selection.