Case context: A machine translation team built a system with an RNN encoder-decoder producing ScoreA(S) and a beam search that outputs the sentence Sout. For a batch of mistranslated sentences, they computed ScoreA(S*) for the human-generated reference and ScoreA(Sout) for the system's output. In the majority of these error cases, ScoreA(S*) turned out to be greater than ScoreA(Sout).

Question: Given this pattern across the error cases, what should the team conclude, and what should they try next to address the errors?

Sample answer: Since ScoreA(S*) is greater than ScoreA(Sout) in most of the error cases, this indicates a search algorithm problem: the scoring function is correctly ranking the human translation higher, but beam search is failing to find the value of S that maximizes ScoreA(S). The team should focus their efforts on improving the search algorithm rather than the scoring function, for example by increasing the beam width used in beam search.

Key points:

• Pattern of ScoreA(S*) > ScoreA(Sout) indicates a search algorithm problem
• The scoring function is functioning correctly in this scenario
• Effort should be focused on the search algorithm, not the scorer
• Increasing beam width is a concrete suggested fix

Rubric: Full credit: correctly identifies the pattern as a search algorithm problem and recommends improving search (e.g., wider beam width), not the scorer. Partial credit: identifies the correct diagnosis but omits the recommended action, or vice versa. No credit: misdiagnoses the issue as an objective/scoring function problem.