Describe how training error, generalization, and target-task performance are evaluated in a four-dataset framework.
Question: When training and dev/test data come from different distributions, we use a four-dataset evaluation framework. Analyze the distinct evaluation purposes of the training set, the training dev set, and the dev and/or test sets, explaining how each helps diagnose the algorithm's performance.
Sample answer: The four-dataset framework evaluates different aspects of performance. The training set is used to evaluate training error. The training dev set evaluates the algorithm's ability to generalize to new data drawn from the training set distribution. Finally, the dev and/or test sets evaluate the algorithm's performance on the task you care about.
Key points:
- Evaluating on the training set measures the training error.
- Evaluating on the training dev set measures generalization to new data drawn from the training set distribution.
- Evaluating on the dev and/or test sets measures performance on the target task you care about.
Rubric: The candidate must explain the specific evaluation purpose for each of the three groups: training set (training error), training dev set (generalization to training set distribution), and dev/test sets (performance on the target task).
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Which dataset is used to evaluate an algorithm's ability to generalize to new data drawn from the training set distribution?
Training error is evaluated by running the algorithm on the training set.
The algorithm's performance on the task you care about is evaluated using the _____ and/or test sets.
Match each dataset in the four-dataset framework to its primary evaluation purpose.
Order the four dataset evaluations from measuring training error through to final target-task performance as described in ML Yearning.
According to ML Yearning, which dataset combination evaluates the algorithm's performance on the task you care about?
The training dev set is drawn from the same distribution as the dev and test sets.
In the four-dataset framework, the _____ dev set evaluates generalization to new data drawn from the training distribution.
Match each evaluation goal to the dataset that measures it in ML Yearning's four-dataset framework.
Order the diagnostic reasoning steps a practitioner follows when using the four-dataset framework to identify an algorithm's key weakness.
Describe how training error, generalization, and target-task performance are evaluated in a four-dataset framework.