While Alignment-based Phrase Translation Evaluation (APT-Eval) measures idiom translation accuracy, it cannot detect literal translation errors in inaccurate translations. In contrast, the Literal Translation Error Rate (LiTER) metric is specifically designed to identify and measure these literal translation errors.

University of California, Berkeley

Google

Literal Translation Error Rate (LiTER) is a metric for a model’s literal translation errors. A literal translation error happens when a sentence is wrongly translated literally in the target language. Translation errors are identified by checking if a translation output contains any (automatically created compiled) blocklisted words. 


Automatic Targeted Evaluation 


Alignment-based Phrase Translation Evaluation (APT-Eval) extends Fadee et al. (2018) and subword-level metrics to measure idiom translation accuracy. It uses word alignments to find words in the hypothesis and reference sentences which align with the annotated idiom source span and compares the two. 

Alignment-based Phrase Translation Evaluation (APT-Eval) 


A blocklist is a list of candidate literal translation errors. 


Blocklist Definition  


1. Select from the source text the list of words $$s = \langle s_1, s_2, \dots, s_N \rangle$$ belonging to the annotated idiom.
2. For each word $$s_i$$, obtain its word translations in the target language using a bilingual dictionary and add them to a blocklist $$b_i = \langle t_1, t_2, \dots, t_M \rangle$$ and create a candidate list of blocklists $$B_s = \langle b_1, b_2, \dots, b_N \rangle$$.
3. For each word in the reference (R), search if it occurs in any of the blocklists $$b_i$$. If so, remove the corresponding blocklist $$b_i$$ from $$B_s$$.
4. Check if the hypothesis contains any blocklisted words. If so, mark this hypothesis as having a literal translation error.

The final score is the percentage of translations that trigger the blocklist. Since LiTER requires source-side annotations, test data with idioms on the source side are collected, and the spans where they occur are annotated.

LiTER Algorithm

LiTER vs. APT-Eval

APT-Eval metrics consist of unigram precision which measures the ratio of words in the reference spans occurring in the hypothesis spans; seen in Fadaee et al. (2018). ChrF (Popovic, 2015) measures character n-gram overlap 


Learn Before

Related