Bilingual word dictionaries are referenced for translating the words of an annotated source span, producing blocklists with candidate literal translation errors. Blocklists are filtered by removing words that occur in the reference to avoid triggering the blocklist when the correct translation is literal. 


Blocklist Creation


A blocklist is a list of candidate literal translation errors. 


University of California, Berkeley

Literal Translation Error Rate (LiTER) is a metric for a model’s literal translation errors. A literal translation error happens when a sentence is wrongly translated literally in the target language. Translation errors are identified by checking if a translation output contains any (automatically created compiled) blocklisted words. 


Automatic Targeted Evaluation 


Blocklist Definition  


- (1) Select from the source text the list of words $s = <s_1, s_2, …, s_N>$ belonging to the annotated idiom.  
- (2) For each word $s_i$, obtain its word translations in the target language using a bilingual dictionary and add them to a blocklist $b_i = <t_1, t_2, …, t_M>$ and create a candidate list of blocklists $B_s = <b_1, b_2, …, b_N>$ 
- (3) For each word in the reference (R), search if it occurs in any of the blocklists $b_i$. If so, remove corresponding blocklist $b_i$ from $B_s$. 
- (4) Check if the hypothesis contains any blocklisted words. If so, mark this hypothesis as having a literal translation error. 
- The final score is the percentage of translations that trigger the blocklist. Since LiTER requires source=side annotations, test data with idioms on the source side are collected, and the spans they occur are annotated. 

Learn Before

Related

Learn After