Why more data alone cannot solve high training error
Question: In a concise analytical response, explain why adding more training examples cannot by itself deliver the target performance when the model already performs too poorly on the training set.
Sample answer: If the model's training error is already above the target, more data alone will not push it below that target. With additional training examples, training error typically stays flat or rises slightly, because the model is not being changed to fit the current data better. Also, development error is usually worse than training error. So if the model cannot reach the goal on training data, it is even less likely to reach it on development data by adding data alone; the underlying model or training procedure must also improve.
Key points:
- The training-set error starts above the desired level.
- Adding more data does not normally reduce training error.
- Development error is usually higher than training error.
- Therefore, more data by itself is not enough to reach the target.
Rubric: A strong response identifies the starting gap on training data, notes that training error does not fall just because more data is added, states that development error is usually higher, and concludes that another change besides data is needed.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why More Data May Not Fix a High-Training-Error Model
Adding more training examples by itself can guarantee that dev error will reach a target level even when training error is still above that target.
When the training set gets larger, the training error can only stay the same or _____.
Match each learning-curve element to its role in diagnosing when extra data will not solve the problem.
Order the logic that shows more data by itself will not achieve the target performance.
Why more data alone cannot solve high training error
Decide whether more data alone can close a large training gap.
Why can’t adding more data by itself solve this learning-curve problem?
Which learning-curve pattern most clearly shows that collecting more data by itself will not solve the problem?
If training error does not improve as more labeled examples are added, development error can still be expected to drop below a lower target level from extra data alone.