How should you respond to search errors versus score-estimation errors in an inference system?
Question: A ranking system produces a bad result. Sometimes the problem is that the search procedure never considered the best candidate. Other times the problem is that the scoring model gave poor estimates to candidates it did consider. Explain how the fix differs in these two cases.
Sample answer: The response depends on which part failed. If the search procedure is at fault, the main task is to improve the search method so it can find better candidates or explore the space more effectively. If the score estimates are the problem, the focus should shift to improving the model or learning process that produces those scores, because the issue is not candidate selection but inaccurate evaluation.
Key points:
- A search error means the system failed to reach a high-scoring candidate even though one existed.
- A score-estimation error means the candidates were evaluated poorly.
- Search failure calls for better search.
- Score failure calls for better training or scoring.
Rubric: A correct answer must distinguish the two failure modes and state the appropriate remedy for each one. It should say that search problems are fixed by improving the search procedure, while scoring problems are fixed by improving the model or training process that assigns scores.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Search-Procedure Error in Inference
Detecting a Problem in the Scoring Step
Score Comparison Diagnostic for Search Errors
When a route-planning system produces a poor route, which two possible failure sources should you separate during debugging?
Diagnosing whether the candidate generator or the reranker caused a missed result helps you decide where to improve a two-stage retrieval system first.
If the approximate search procedure cannot find the value of S that gives the best Score_A(S), the right fix is to improve the _____ procedure.
Match each inference failure type to its correct description in a two-stage prediction system.
Put the debugging steps for an incorrect inference result in the right order.
What is a scoring function problem when diagnosing inference behavior in a machine learning system?
When an inference system gives poor results, randomly choosing between improving retrieval and improving the scoring model is a good debugging strategy.
Choose the missing word in the guidance about fixing a scoring-function problem.
Match each failure cause to the action that best addresses it.
Order the checks for deciding whether a bad prediction comes from the scorer rather than the search process.
How should you respond to search errors versus score-estimation errors in an inference system?
Diagnosing an Incorrect Text Completion from Search Scoring
Why a coin flip is a bad debugging strategy for inference failures