Score Comparison Diagnostic for Search Errors
This diagnostic compares the score of a trusted reference answer with the score of the system’s answer for the same input. If the trusted answer receives the better score, the main problem is likely in the search or optimization procedure. If not, the scoring rule or its implementation is the more likely source of error.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Search-Procedure Error in Inference
Detecting a Problem in the Scoring Step
Score Comparison Diagnostic for Search Errors
When a route-planning system produces a poor route, which two possible failure sources should you separate during debugging?
Diagnosing whether the candidate generator or the reranker caused a missed result helps you decide where to improve a two-stage retrieval system first.
If the approximate search procedure cannot find the value of S that gives the best Score_A(S), the right fix is to improve the _____ procedure.
Match each inference failure type to its correct description in a two-stage prediction system.
Put the debugging steps for an incorrect inference result in the right order.
What is a scoring function problem when diagnosing inference behavior in a machine learning system?
When an inference system gives poor results, randomly choosing between improving retrieval and improving the scoring model is a good debugging strategy.
Choose the missing word in the guidance about fixing a scoring-function problem.
Match each failure cause to the action that best addresses it.
Order the checks for deciding whether a bad prediction comes from the scorer rather than the search process.
How should you respond to search errors versus score-estimation errors in an inference system?
Diagnosing an Incorrect Text Completion from Search Scoring
Why a coin flip is a bad debugging strategy for inference failures
Learn After
Classifying development-set mistakes with optimization verification
Approximate Scoring Followed by Approximate Search
Checking Whether a Translation Error Comes from Search or Scoring
Checking Reward Quality in Reinforcement Learning
A useful human output for optimization verification
What does this optimization check suggest when the reference candidate scores higher than the algorithm’s output?
Using the Optimization Verification Test
Optimization Check Symbols
How to carry out an optimization verification check
Using the Optimization Verification Check
Debugging a Translation Decoder
When to Attribute an Error to Optimization
When Is an Optimization Check Useful?
Checking the Source of a Performance Problem
Optimization checks can isolate the source of a training problem