Learn Before
Choose the right size for a review set in an image recognition project.
Case context: You are training a system to identify types of flowers in smartphone photos. People can usually tell the classes apart without difficulty, so your team set up a small review set for manual inspection. After checking a handful of mistakes, the sample is too small to reveal any clear recurring failure patterns.
Question: What should your team conclude about the size of this review set, and what should they do next so it serves its purpose?
Sample answer: The team should conclude that the review set is too small and expand it. Its job is to provide a representative sample of the model's main error categories, so if only a few mistakes are available and no patterns appear, the set is not large enough for useful manual analysis.
Key points:
- Diagnose the review set as too small to expose recurring error categories.
- Increase the size of the review set.
- Link the decision to finding the model's major error categories in a task people handle well.
Rubric: The response must state that the current review set is insufficient and should be enlarged. It must explain that the purpose of enlarging it is to better surface the model's main error categories in a human-easy classification task.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
A Small Set of Mistakes Can Still Guide Priorities
A Small Sample of Dev Errors Can Reveal Major Failure Patterns
Reviewing About 50 Mistakes Reveals Main Error Sources
Reviewing About 100 Eyeball Dev Errors Usually Reveals the Main Failure Patterns
Lower Error Rates Require Larger Review Sets
Why should the manually reviewed dev subset be large enough?
The rough sizing guidance for an eyeball development set is meant for tasks that people can evaluate reliably.
A validation sample should be large enough to reveal the system's main _____.
Match each example count to the amount of insight it typically provides when reviewing errors by hand.
Order the steps for reviewing a validation set to find the most common model mistakes.
Which example task is used to illustrate a rough Eyeball dev set size recommendation?
A tiny validation sample is enough to uncover every major failure mode in a model.
When manual error review is practical
Match each concept to the best description in a model error-analysis setting.
Decide Whether a Human Review Set Is Large Enough
How should an Eyeball dev set be sized to reveal the main error patterns?
Choose the right size for a review set in an image recognition project.
Purpose of an Eyeball Dev Set for Human-Level Tasks