What the training dev set tells you
Question: In a four-dataset evaluation setup, what does performance on the training dev set tell you about a learning algorithm?
Sample answer: It measures how well the algorithm generalizes to new examples that come from the same distribution as the training set.
Key points:
- It checks generalization to unseen data.
- The unseen data should match the training-set distribution.
Rubric: The answer must say that the training dev set evaluates generalization to new data drawn from the training-set distribution.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Which dataset best measures how well a model handles new examples from the same distribution as its training data?
Training error is computed by evaluating the model on the same examples used to fit it.
Which Set Measures the Performance You Care About?
Match each dataset in the four-way evaluation setup to what it is mainly used to measure.
Order the evaluations used to move from training fit to the final assessment of performance on the target distribution.
Which dataset or datasets should be used to estimate how well a model will perform on the distribution you ultimately care about?
A training-dev set should come from the same distribution as the dev and test sets so it measures performance in the target environment.
Training-Distribution Development Set
Connect Each Dataset to Its Evaluation Purpose
Using the Four-Dataset Framework to Isolate the Main Weakness
How do the different datasets in a four-set evaluation scheme serve different diagnostic purposes?
What each split shows in a four-way classifier check
What the training dev set tells you