Question: Suppose your machine learning system outputs $y_{out}$ instead of the correct response $y^*$. Describe how you would use the Optimization Verification test to diagnose the problem, and explain the two possible outcomes and what they mean for your system's components.

Sample answer: To use the Optimization Verification test, I would compute the score for the correct response, $Score_x(y^*)$, and the score for the system's output, $Score_x(y_{out})$. Then, I would compare the two scores. If $Score_x(y^*) > Score_x(y_{out})$, it means the scoring function correctly assigned a higher score to the right answer, but the search/optimization algorithm failed to find it. In this case, I would blame the optimization algorithm. If $Score_x(y^*) \le Score_x(y_{out})$, it means the scoring function preferred the incorrect answer over the correct one, even though the correct one was available. In this case, I would blame the scoring function computation.

Key points:

• Compute $Score_x(y^*)$ and $Score_x(y_{out})$.
• Compare the two scores.
• If $Score_x(y^*) > Score_x(y_{out})$, blame the optimization or search algorithm.
• If $Score_x(y^*) \le Score_x(y_{out})$, blame the scoring function computation.

Rubric: A good response will describe the process of computing and comparing the two scores and accurately interpret both possible outcomes of the inequality.