Question: A prediction system produces $z_{pred}$ instead of the correct output $z_{true}$. Explain how you would apply the Optimization Verification check to investigate the issue, and describe the two possible outcomes and what each one suggests.

Sample answer: I would first evaluate the scoring function on the correct output, $Score_x(z_{true})$, and on the system’s output, $Score_x(z_{pred})$. Then I would compare the two values. If $Score_x(z_{true}) > Score_x(z_{pred})$, the score function correctly ranks the true answer higher, which means the search or optimization procedure failed to find it. In that case, the likely problem is the optimization algorithm. If $Score_x(z_{true}) \le Score_x(z_{pred})$, the score function itself prefers the wrong output, even though the correct one was available. In that case, the scoring function computation is the more likely source of the error.

Key points:

• Compute $Score_x(z_{true})$ and $Score_x(z_{pred})$.
• Compare the two scores.
• If $Score_x(z_{true}) > Score_x(z_{pred})$, blame the optimization or search procedure.
• If $Score_x(z_{true}) \le Score_x(z_{pred})$, blame the scoring function computation.

Rubric: A strong response explains how to compute both scores, compares them clearly, and interprets each inequality correctly in terms of whether the problem is in optimization or in scoring.