In practice, apply the Optimization Verification test across errors in the dev set. Each error where the correct output scores higher is marked as an optimization-algorithm error; each error where it does not is counted as a scoring-function error.

Applying Optimization Verification Across Dev Set Errors

A common AI design pattern is to first learn an approximate scoring function and then use an approximate maximization algorithm. Recognizing this pattern lets one use Optimization Verification to understand the source of errors.

Approximate Scoring Function Plus Approximate Maximization Pattern

For machine translation, compute a score for possible translations and use heuristic search because the set of possible English sentences is too large. If the correct translation scores above the system translation, attribute the error to approximate search; otherwise, attribute it to the score computation.

Optimization Verification for Machine Translation

For a reinforcement learning system whose trajectory is worse than a human pilot trajectory, compare the reward assigned to the human trajectory with the reward assigned to the algorithm trajectory. If the human trajectory scores higher, improving the reinforcement learning algorithm is worthwhile; if it does not, improve the reward function.

Optimization Verification for Reinforcement Learning Reward Functions

Optimization Verification does not require a truly optimal output. If a superior output is available, such as a human pilot trajectory that is better than the current learning algorithm output, the test can still indicate whether improving the optimization algorithm or the scoring function is more promising.

Using a Superior Human Output in Optimization Verification

The Optimization Verification test compares the score of a known correct output with the score of the system output. If the correct output scores higher, blame the optimization or search algorithm; otherwise, blame the scoring function computation.

Google

When an inference system returns a wrong output, debugging should distinguish whether the approximate search algorithm failed or whether the scoring function assigned inaccurate scores. Knowing which cause is responsible determines whether to improve the search algorithm or the learning algorithm that estimates the score.

Debugging Inference Algorithms

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A search algorithm problem occurs when the scoring function assigns a higher score to the correct output, but the approximate search algorithm still selects the wrong output. This indicates that improving the search algorithm, such as increasing beam width in beam search, is the right focus.

Search Algorithm Problem in Inference

A scoring function problem occurs when the scoring computation fails to give the correct output a strictly higher score than the incorrect system output. In this case, the useful work is to improve how the system learns or approximates the score.

Scoring Function Problem in Inference

Optimization Verification Test

When an inference system returns a wrong output, which two root causes should a developer distinguish between to debug effectively?

Identifying whether the search algorithm or scoring function caused an inference failure determines which component you should prioritize improving.

If the approximate search algorithm fails to find the value of S that maximizes Score_A(S), the correct fix is to work on improving the _____ algorithm.

Match each inference failure type to its precise description as defined in Machine Learning Yearning.

Arrange the steps of the inference debugging process described in MLY in the correct logical order.

According to MLY, what is a 'scoring function problem' in the context of debugging inference systems?

Randomly choosing whether to improve the search algorithm or scoring function is an acceptable debugging strategy when an inference system produces wrong outputs.

When the scoring function is identified as the inference failure cause, you should work on the _____ algorithm that estimates Score_A(S).

Match each identified inference failure cause to the correct recommended action from Machine Learning Yearning.

Arrange the reasoning steps used to determine whether an inference failure is caused by an inaccurate scoring function.

Question:
Analyze the two primary failure modes of an inference system: a search algorithm problem and a scoring function (objective) problem. Contrast how the target actions you must take to improve the system differ depending on which of these two issues is identified as the root cause of a wrong output.

Sample answer:
When debugging an inference system that yields incorrect output, you must distinguish between a search algorithm failure and a scoring function failure. If the root cause is a search algorithm problem (where the approximate search failed to find the value of S that maximizes the score), the developer must prioritize improving the search algorithm itself. Conversely, if the root cause is a scoring function problem (where the estimated scores themselves are inaccurate), the developer must focus on improving the learning algorithm that estimates the scores rather than modifying the search algorithm.

Key points:
- A search algorithm problem means the approximate search failed to find the value that maximizes the score.
- A scoring function problem means the estimates for the scores were inaccurate.
- If the search algorithm failed, priority must go to improving the search algorithm.
- If the scoring function failed, priority must go to improving the learning algorithm that estimates the score.

Rubric:
The answer must accurately define the actions for both failure modes. It should clearly state that a search algorithm problem requires working on the search algorithm itself, while a scoring function (objective) problem requires working on the learning algorithm that estimates the score.

Contrast the target actions required when debugging an inference system's search algorithm versus its scoring function.

Case context:
You are building a speech-to-text inference system. For a given audio input A, the system outputs transcription S_out, which is incorrect. The target correct transcription is S_star. The system uses beam search to approximate finding a transcription S that maximizes ScoreA(S). To diagnose the failure, you compute the scores and find that ScoreA(S_star) is greater than ScoreA(S_out), but the beam search returned S_out.

Question:
Based on the provided scoring relationship, diagnose whether this is a search algorithm problem or a scoring function problem, and decide which development effort you should prioritize to fix this error.

Sample answer:
Since ScoreA(S_star) is greater than ScoreA(S_out), the scoring function correctly assigned a higher score to the correct transcription S_star. However, the system still output S_out. This means the approximate search algorithm (beam search) failed to find the value S that maximizes the score, which is a search algorithm problem. Therefore, you should prioritize improving the search algorithm itself rather than the learning algorithm that estimates the score.

Key points:
- Identify that the scoring function correctly preferred the true output over the returned output because ScoreA(S_star) > ScoreA(S_out).
- Conclude that the beam search failed to find the higher-scoring correct transcription, which is a search algorithm problem.
- Decide to prioritize improving the search algorithm (beam search) rather than the learning algorithm.

Rubric:
The response must identify the issue as a search algorithm problem based on the fact that the correct answer had a higher score but was not selected. It must also specify that the priority should be to improve the search algorithm.

Diagnosing incorrect transcriptions in a speech recognition inference system

Question:
According to Machine Learning Yearning, why is it problematic for researchers to randomly decide whether to work on the search algorithm or the scoring function when debugging an inference failure?

Sample answer:
Randomly choosing which component to improve is problematic because the two causes of failure (search algorithm vs. scoring function) require completely different engineering responses. Working on the wrong component will not resolve the underlying failure, resulting in wasted development effort.

Key points:
- The two possibilities of failure require prioritizing efforts very differently (improving search vs. improving the learning algorithm).
- Randomly choosing can lead to working on the wrong component, which will not fix the actual bug.

Rubric:
The answer should explain that the two failure types require completely different priorities/actions, and randomly choosing can lead to prioritizing the wrong algorithm, wasting effort without fixing the issue.

Learn Before

Related

Learn After