LiTER aims to help produce models that improve on LitTER without sacrificing general quality. LiTER is expected to be relevant when developing NMT models for creative content like subtitles, social media text. 


LiTER’s Goal


1. Block-listed words appear appear as a result of translating non-idiomatic words count as an error 
2. Experiments were conducted on languages with simple morphology. LitTER struggles with compound words (e.g. in German) and have been addressed with custom rules
3. This metric is ambiguous when used in isolation and should be paired with standard evaluation metrics when scoring translation models 

LitTER Limitations 


Literal Translation Error Rate (LiTER) aims to complement global evaluation metrics by providing insights through targeted evaluation. It should not be used in isolation, as a lower LiTER score could result from either more accurate idiom translations or from model hallucinations.

University of California, Berkeley

Google

1. Select from the source text the list of words $$s = \langle s_1, s_2, \dots, s_N \rangle$$ belonging to the annotated idiom.
2. For each word $$s_i$$, obtain its word translations in the target language using a bilingual dictionary and add them to a blocklist $$b_i = \langle t_1, t_2, \dots, t_M \rangle$$ and create a candidate list of blocklists $$B_s = \langle b_1, b_2, \dots, b_N \rangle$$.
3. For each word in the reference (R), search if it occurs in any of the blocklists $$b_i$$. If so, remove the corresponding blocklist $$b_i$$ from $$B_s$$.
4. Check if the hypothesis contains any blocklisted words. If so, mark this hypothesis as having a literal translation error.

The final score is the percentage of translations that trigger the blocklist. Since LiTER requires source-side annotations, test data with idioms on the source side are collected, and the spans where they occur are annotated.

Learn Before

Related

Learn After