Learn Before
Analyzing the Cause and Solution for Dev Set Overfitting
Question: Explain how the process of repeatedly evaluating ideas during development can lead to dev set overfitting. Furthermore, describe how you would diagnose this issue at the end of development and what specific action you should take to resolve it.
Sample answer: Repeatedly evaluating ideas on the dev set causes the algorithm to gradually overfit to the specific characteristics of that dev set. This is diagnosed at the end of development by comparing performance on the dev set versus the test set; if dev set performance is significantly better, overfitting has occurred. To resolve this, you should discard the overfit dev set and get a fresh dev set.
Key points:
- Repeated evaluation on the dev set causes gradual overfitting.
- Diagnosis requires comparing dev set performance against test set performance after development.
- A large gap where dev set performance is better indicates overfitting.
- The recommended solution is to get a fresh dev set.
Rubric: The student should clearly identify the cause (repeated evaluation), the diagnosis method (comparing dev and test set performance), and the solution (getting a fresh dev set).
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Avoiding Test Set Decisions During Regular Progress Tracking
What causes an algorithm to gradually overfit to the dev set during the development process?
If dev set performance is much better than test set performance after development, this is a sign of dev-set overfitting.
When dev set performance is much better than test set performance, Machine Learning Yearning recommends you get a _____ dev set.
Match each concept related to dev-set overfitting to its correct description.
Arrange the steps of the full dev-set overfitting lifecycle in the correct order.
After development, your dev set performance is far better than your test set performance. What does Machine Learning Yearning recommend?
According to Machine Learning Yearning, you should regularly evaluate your algorithm on the test set throughout development to track progress.
The process of repeatedly evaluating ideas on the dev set causes your algorithm to gradually _____ to the dev set.
Match each verbatim phrase from Machine Learning Yearning to what it refers to or signifies.
Arrange the reasoning steps for diagnosing and responding to a dev-test performance gap in the correct order.
Analyzing the Cause and Solution for Dev Set Overfitting
Diagnosing a Performance Gap After Development
Identifying the Indicator of Dev Set Overfitting