Learn Before
Plan the next review step after repeated image-classifier mistakes.
Case context: A team is developing a model that recognizes road signs from dashcam images. During manual review of dev set errors, they find that many mistakes come from images that are heavily blurred by motion. The team does not yet know a practical method for fixing blur-related failures. Some people want to stop discussing those examples and focus only on error types they already know how to address.
Question: According to sound error-analysis practice, what should the team do about the blurred images, and why?
Sample answer: They should keep the blurred images in the analysis. The team should count how often blur appears among the errors and examine those examples carefully, even if there is no immediate fix. The purpose of reviewing mistakes is to discover where the system is weak and to uncover promising directions for improvement, not only to track problems that are already easy to solve. Looking closely at the blurred cases may suggest new ideas, and measuring their frequency helps the team decide whether a dedicated effort to reduce blur-related errors is worth pursuing.
Key points:
- Do not exclude errors just because they are hard to fix right away.
- Error analysis is meant to reveal promising improvement opportunities.
- Reviewing the blurred examples may lead to new ideas.
- Counting how common blur is helps with prioritization.
Rubric: The response should advise keeping the blurred images in the error analysis, explain that the review is meant to identify promising directions rather than only easy fixes, and note that the frequency of blur-related errors helps the team decide whether to invest in a targeted solution.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
One Example May Fit Several Error Tags
New Error Categories Can Appear During Review
Choose Error Categories You Can Act On
Error Review Improves Through Repeated Passes
Using Error Counts to Decide Where to Focus Next
Working on Several Error Buckets at Once
Error Analysis Is Not an Automatic Ranking Rule
A Category's Share of Errors Sets an Upper Bound on Improvement
Error Analysis Helps Estimate Whether a Proposed Change Is Worth the Effort
Why Quick Error Review Is Often Skipped
Incorrect Labels in a Validation Set
Splitting a large development set into a review subset and a tuning subset
Build a Simple Baseline First, Then Use Error Analysis to Prioritize Improvements
Using Training-Set Mistakes to Diagnose High Bias
Reviewing a Sample of Validation Errors
Separating Search Errors from Scoring Errors in Inference
Component-Wise Error Review
Error Analysis as a Data-Science Lens on Model Mistakes
Multiple Valid Approaches to Error Analysis
Tasks Humans Can Perform Give Stronger Error Analysis Benchmarks
When diagnosing a model, what should error analysis focus on first?
Error analysis on a machine learning system must follow one fixed procedure.
Name the practice of reviewing mistakes to understand why predictions failed.
Match each error-analysis idea to the description that best fits it.
Put the steps of a simple dev-set error review in the right order.
Why is it useful to inspect misclassified examples during error analysis, even for error types you cannot immediately repair?
Error analysis is usually repeated after each round of model changes.
Error analysis can help you judge which improvement paths look most _____.
Match each error-analysis activity with the benefit it can provide.
Order the steps for deciding which error types to target after a first pass of error review.
Why Error Review Helps Set the Right Next Priorities
Plan the next review step after repeated image-classifier mistakes.
What is error analysis used for in machine learning?