Literal Translation Error Rate (LiTER) is a metric for a model’s literal translation errors. A literal translation error happens when a sentence is wrongly translated literally in the target language. Translation errors are identified by checking if a translation output contains any (automatically created compiled) blocklisted words. 


Automatic Targeted Evaluation 


Alignment-based Phrase Translation Evaluation (APT-Eval) extends Fadee et al. (2018) and subword-level metrics to measure idiom translation accuracy. It uses word alignments to find words in the hypothesis and reference sentences which align with the annotated idiom source span and compares the two. 

Alignment-based Phrase Translation Evaluation (APT-Eval) 


Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

Standard evaluation methods in neural machine translation (NMT) often overlook the varying frequencies of different idioms, potentially favoring models overfitted on over-represented expressions. To address this issue in targeted evaluation, performance scores are macro-averaged over individual idioms rather than micro-averaged over all occurrences.

Macro-Averaging to Address Varying Idiom Frequencies

Targeted evaluation isolates the idiom translation with word alignments or word edit distance, which measures idiom translation accuracy but doesn’t account for literal translation errors. Shao et al. (2018) proposed a method for estimating these errors’ frequency but it requires hand-crafted blocklists with words corresponding to literal translation errors. 


University of California, Berkeley

The translation of idiomatic expressions such as “under the weather” is a current problem in Neural Machine Translation. NMT models incorrectly translate idiomatic expressions’ meanings literally or word-by-word. 

Translation of Idiomatic Expressions 


The lack of targeted and automatic methods for quantifying these errors limits current research on translating idiomatic expressions. 


Obstacles for idiomatic expressions in NMT 


Global Metrics (e.g. BLEU) consider the full translation and obscure the effects of idiom translation


Global metrics for NMT of idiomatic expression 


Targeted Evaluation for NMT of idiomatic expression 


Related work in Idioms in NMT and Targeted Evaluation. 


Learn Before

Related

Learn After