Learn Before
Error Category Fraction as a Ceiling on Possible Error Reduction
The fraction of misclassified examples that belong to a particular category is a ceiling—the maximum possible amount—on how much addressing that category alone could reduce the system's errors. For example, if only 5% of the misclassified images are dogs, then no matter how much dog-image performance is improved, no more than 5% of the errors can be removed, taking a 10% error rate down to at best about 9.5%. By contrast, if 50% of the mistakes are dogs, addressing them could potentially cut the error rate in half, for instance from 10% down to 5%.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Examples Can Belong to Multiple Error Categories
Discovering New Error Categories While Reviewing Examples
Most Helpful Error Categories Are Ones You Have Ideas to Improve
Error Analysis Is an Iterative Process
Error Category Frequency Helps Indicate Which Categories to Focus On
Pursuing Multiple Error Categories in Parallel
Error Analysis Does Not Yield a Rigid Priority Formula
Error Category Fraction as a Ceiling on Possible Error Reduction
Error Analysis as a Quantitative Basis for Project Investment Decisions
Engineer Reluctance to Perform Error Analysis Despite Its Low Time Cost
Mislabeled Examples in the Dev Set
Splitting a Large Dev Set into a Manually Examined Subset and a Hands-Off Subset
Build a Basic System Quickly and Iterate Using Error Analysis
Training Set Error Analysis for High Bias
Manually Reviewing 100 Speech Recognition Dev Set Examples
Debugging Inference Algorithms
Error Analysis by Parts
Error Analysis as Data Science for ML Mistakes
No Single Right Way to Perform Error Analysis
Human-Solvable Problems Enable More Powerful Error Analysis Tools
What does error analysis primarily examine to understand an ML system's mistakes?
There is exactly one correct method for conducting error analysis on an ML system.
The process of looking at misclassified examples to understand error causes is called _____.
Match each error analysis concept to its correct description from Machine Learning Yearning.
Order the steps of conducting a basic error analysis on a dev set as described in Machine Learning Yearning.
What is the primary goal of reviewing misclassified examples during error analysis, even in categories you cannot yet fix?
Machine Learning Yearning describes error analysis as an iterative process.
Error analysis can often help you figure out how _____ different improvement directions are.
Match each error analysis activity to the benefit it provides according to Machine Learning Yearning.
Order the reasoning steps for deciding which error categories to pursue after completing an initial error analysis.
Learn After
What does the error category fraction represent as a 'ceiling' in error analysis?
If only 5% of misclassified images are dogs, improving dog recognition will definitively remove 5% of all errors.
The fraction of misclassified examples in a category is a _____ on how much fixing that category can reduce total errors.
Match each error category fraction to its correct implication for a system with a 10% error rate.
Order the steps for applying error category fractions to decide whether a category is worth prioritizing.
A classifier has 10% error and 50% of its mistakes are dog images. What is the best possible accuracy after perfectly fixing dog classification?
A high error category fraction guarantees that fixing that category will substantially improve overall system performance.
With a 10% error rate and 5% of mistakes being dogs, the best achievable error rate after fixing dogs perfectly is _____.
Match each key term from error-ceiling analysis to its correct definition.
Order the reasoning steps for interpreting an error category fraction and evaluating that category's priority.