Why inspecting a small set of bad training cases can reveal training-set problems
Question: Explain why looking through roughly 100 training examples that a model handles badly can reveal the main kinds of training-set mistakes, and how this resembles error analysis on a development set.
Sample answer: Reviewing about 100 low-performing training examples gives a direct view of what is going wrong, instead of only seeing a single overall loss or accuracy number. By sorting those cases into a few error categories, a practitioner can see which problems occur most often and which data issues deserve attention first. This is the same basic idea used in development-set error analysis: classify mistakes by type, count how often each type appears, and use those counts to decide what to fix.
Key points:
- Inspect roughly 100 examples the model gets wrong
- Look for recurring categories of training-set errors
- Count examples in each category to find the biggest problems
- The method is analogous to dev-set error analysis
Rubric: Full credit requires mentioning: (1) reviewing about 100 examples, (2) identifying major categories of errors, and (3) the link to counting errors in a way similar to dev-set analysis.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Common Error Labels for Volunteer Audio Clips
What should you do first when a model is making many mistakes on the training set?
True or False: A useful way to inspect training mistakes is to group them into categories and count how often each type appears, similar to dev-set error analysis.
How many bad training examples should you inspect first?
Match Each Training-Review Idea to Its Description
Put the training-set review process in the correct order.
Why inspecting a small set of bad training cases can reveal training-set problems
Diagnosing Poor Training Performance in a Speech Model
What should you do when training performance is poor?
What is the main purpose of reviewing training examples that the model handles poorly?
True or False: Diagnosing model failures requires checking every training example in the dataset.