1. Block-listed words appear appear as a result of translating non-idiomatic words count as an error 
2. Experiments were conducted on languages with simple morphology. LitTER struggles with compound words (e.g. in German) and have been addressed with custom rules
3. This metric is ambiguous when used in isolation and should be paired with standard evaluation metrics when scoring translation models 

University of California, Berkeley

Literal Translation Error Rate (LiTER) aims to complement global evaluation metrics by providing insights through targeted evaluation. It should not be used in isolation, as a lower LiTER score could result from either more accurate idiom translations or from model hallucinations.

LiTER as a Complementary Metric

LitTER Limitations 


LiTER's goal is to guide the development of NMT models that reduce literal translation errors (i.e., achieve lower LiTER scores) without sacrificing overall translation quality. This goal is especially relevant when developing models for creative content such as subtitles and social media text, where idiomatic expressions are common.

Learn Before

Related