Case context: A language lab trained an encoder-decoder model to translate short product-support queries. The model assigns a score H(t) to any candidate translation t, and the decoder uses beam search to choose an output called t_hat. The researchers reviewed 18 mistranslations and compared the score of each human reference t_ref with the score of the system output t_hat. In most of those failures, H(t_ref) was higher than H(t_hat).

Question: What does this comparison suggest about the cause of the mistakes, and what change should the team test first?

Sample answer: The pattern suggests a decoding or search problem rather than a scoring problem. Because the reference translation usually receives the higher score, the model appears to know which candidate is better, but beam search is not finding it. The team should therefore improve the search procedure, such as by trying a larger beam size, before changing the scoring model.

Key points:

• When H(t_ref) > H(t_hat) in most failures, the search procedure is likely at fault
• The scoring model is still ranking the better translation higher
• The next step is to improve decoding instead of retraining the scorer
• A practical first experiment is to increase beam size

Rubric: Full credit: correctly identifies a search/decoding problem and recommends improving search, such as using a wider beam, rather than changing the scoring model. Partial credit: gives the right diagnosis but not the next step, or gives the next step without the diagnosis. No credit: claims the main issue is that the scoring function itself is wrong.