Explain why training performance must come before strong validation or test performance.
Question: In a concise analytical response, explain why an algorithm generally needs solid performance on the training set before high validation or test performance is realistic. Use a 78% and 92% example.
Sample answer: Training performance reflects how well the model fits examples it has already seen, while validation or test performance measures how well it generalizes to new examples. In practice, performance on unseen data is usually lower than performance on the training set. So if a model reaches only 78% accuracy on training data, it would be unrealistic to expect 92% accuracy on new data. The model should first improve its training performance before a strong validation or test result is likely.
Key points:
- Training data are examples the model has already seen.
- Validation or test data are new examples the model has not seen.
- Performance on new data is usually lower than performance on training data.
- An 78% training result does not justify expecting 92% on unseen data.
- Better training performance should come first if strong generalization is expected.
Rubric: A strong response distinguishes seen from unseen data, notes that validation or test performance is usually lower, correctly interprets the 78% and 92% example, and concludes that training performance must improve first.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What usually has to happen before a model is expected to do well on validation and test data?
True or False: Performance on development and test data is usually lower than performance on the training set.
Training performance comes first
Interpreting Training and Held-Out Performance
Order the steps used to judge whether a better score on unseen data is believable.
Explain why training performance must come before strong validation or test performance.
Diagnose a claim of 72% training accuracy and 89% dev/test accuracy.
Why must training performance be solid before you expect strong validation or test performance?
Which reported result contradicts the training-first rule?
A model that fits the training set poorly can still be expected to do well on dev or test data.