Learn Before
Separating Search Errors from Scoring Errors in Inference
When an inference system produces an incorrect result, first determine whether the search procedure failed to find a strong candidate or whether the scoring model assigned the wrong values. The next debugging step depends on that diagnosis: improve the search method if candidates are being missed, or improve the model that ranks or scores candidates if the scoring is unreliable.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
One Example May Fit Several Error Tags
New Error Categories Can Appear During Review
Choose Error Categories You Can Act On
Error Review Improves Through Repeated Passes
Using Error Counts to Decide Where to Focus Next
Working on Several Error Buckets at Once
Error Analysis Is Not an Automatic Ranking Rule
A Category's Share of Errors Sets an Upper Bound on Improvement
Error Analysis Helps Estimate Whether a Proposed Change Is Worth the Effort
Why Quick Error Review Is Often Skipped
Incorrect Labels in a Validation Set
Splitting a large development set into a review subset and a tuning subset
Build a Simple Baseline First, Then Use Error Analysis to Prioritize Improvements
Using Training-Set Mistakes to Diagnose High Bias
Reviewing a Sample of Validation Errors
Separating Search Errors from Scoring Errors in Inference
Component-Wise Error Review
Error Analysis as a Data-Science Lens on Model Mistakes
Multiple Valid Approaches to Error Analysis
Tasks Humans Can Perform Give Stronger Error Analysis Benchmarks
When diagnosing a model, what should error analysis focus on first?
Error analysis on a machine learning system must follow one fixed procedure.
Name the practice of reviewing mistakes to understand why predictions failed.
Match each error-analysis idea to the description that best fits it.
Put the steps of a simple dev-set error review in the right order.
Why is it useful to inspect misclassified examples during error analysis, even for error types you cannot immediately repair?
Error analysis is usually repeated after each round of model changes.
Error analysis can help you judge which improvement paths look most _____.
Match each error-analysis activity with the benefit it can provide.
Order the steps for deciding which error types to target after a first pass of error review.
Why Error Review Helps Set the Right Next Priorities
Plan the next review step after repeated image-classifier mistakes.
What is error analysis used for in machine learning?
Learn After
Search-Procedure Error in Inference
Detecting a Problem in the Scoring Step
Score Comparison Diagnostic for Search Errors
When a route-planning system produces a poor route, which two possible failure sources should you separate during debugging?
Diagnosing whether the candidate generator or the reranker caused a missed result helps you decide where to improve a two-stage retrieval system first.
If the approximate search procedure cannot find the value of S that gives the best Score_A(S), the right fix is to improve the _____ procedure.
Match each inference failure type to its correct description in a two-stage prediction system.
Put the debugging steps for an incorrect inference result in the right order.
What is a scoring function problem when diagnosing inference behavior in a machine learning system?
When an inference system gives poor results, randomly choosing between improving retrieval and improving the scoring model is a good debugging strategy.
Choose the missing word in the guidance about fixing a scoring-function problem.
Match each failure cause to the action that best addresses it.
Order the checks for deciding whether a bad prediction comes from the scorer rather than the search process.
How should you respond to search errors versus score-estimation errors in an inference system?
Diagnosing an Incorrect Text Completion from Search Scoring
Why a coin flip is a bad debugging strategy for inference failures