Situation: You are building a machine translation system. For one input sentence, the reference translation is "The parcel arrived on Tuesday" ($S^*$), but your system produces "The parcel arrived on Monday" ($S_{out}$). The model assigns each candidate sentence a value with $Score_A$, and a separate approximate search procedure tries to find the sentence with the best value.

Task: Apply the optimization verification test to determine why the system returned "The parcel arrived on Monday." What values would you compare, and how would those comparisons tell you whether the problem is in the search procedure or in the scoring rule?

Sample answer: First, I would evaluate $Score_A("The parcel arrived on Tuesday")$ and $Score_A("The parcel arrived on Monday")$. Next, I would compare them.

• If $Score_A("The parcel arrived on Tuesday")$ is greater than $Score_A("The parcel arrived on Monday")$, then the scoring rule is favoring the correct translation. In that case, the approximate search procedure is likely at fault because it failed to return the highest-scoring candidate.
• If $Score_A("The parcel arrived on Tuesday")$ is not greater than $Score_A("The parcel arrived on Monday")$, then the scoring rule is giving the wrong answer a score that is too high, so the scoring rule should be corrected.

Key points:

• Measure $Score_A(S^*)$ and $Score_A(S_{out})$.
• Compare $Score_A(S^*)$ with $Score_A(S_{out})$.
• If the correct sentence scores higher, investigate the search procedure.
• If it does not, investigate the scoring rule.

Rubric: A strong answer will name the two candidate sentences, state that both scores should be computed, and explain how the comparison identifies whether the fault lies in search or in scoring.