Focus on the Scoring Model When Most Errors Come from It
If an error audit shows that almost all mistakes come from the scoring component, then improving the optimization procedure can only remove a small fraction of the total errors. In that case, the scoring model is the better place to spend effort.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Focus on the Scoring Model When Most Errors Come from It
When Is a Dev-Set Error Attributed to Optimization?
True or False: If the best achievable development-set score is already no better than the current system's score, the optimization algorithm is the main problem.
A development-set example is counted as a mistake by the _____ metric when it lowers the score used to tune the model.
Which condition indicates that a wrong development-set example is an optimization problem rather than a scoring problem?
If the development set is easier than the test set because the examples come from different sources, improving development accuracy may fail to improve test performance.
When the evaluation score prefers the wrong output, the dev-set issue is a _____ error.
Match each verification expression to its meaning.
Order the steps for deciding whether each development-set mistake points to the optimizer or the scoring rule.
What does Score_A(S*) ≤ Score_A(S_out) indicate in dev set analysis?
A troubleshooting method can start by analyzing one example and then be extended to every error in the development set.
Meaning of T* in an Optimization Check
Match each diagnosis label to its role in separating search problems from scoring problems.
Put the validation analysis steps in the correct order after reviewing example-level errors.
How can you sort validation mistakes into model and metric problems?
Using a Development-Set Check to Separate Search Problems from Metric Problems
When does a verification test point to a scoring problem?
Learn After
A verification check shows that 90% of observed mistakes come from the evaluation metric and 10% come from the optimizer. Where should the next engineering effort go?
If 8% of the model's mistakes come from mislabeled training examples, then fixing the labels perfectly could eliminate at most about 8% of the total mistakes.
If most of the mistakes come from the prediction stage, improve how you make an _____ of the outcome.
Match each error breakdown in a model audit to the improvement implication it suggests.
Order the steps for deciding whether a model improvement effort should target the predictor or the optimizer.
Use an error breakdown to choose between fixing the search routine and improving the scoring model.
How a product team should split effort after checking error sources
Upper Bound from a Small Optimization Error Share
How should improvement work be allocated when different components contribute different shares of the errors?
A component-level error breakdown can indicate the largest fraction of total mistakes that could be removed by perfecting one part of the system.