1. Block-listed words appear appear as a result of translating non-idiomatic words count as an error 
2. Experiments were conducted on languages with simple morphology. LitTER struggles with compound words (e.g. in German) and have been addressed with custom rules
3. This metric is ambiguous when used in isolation and should be paired with standard evaluation metrics when scoring translation models 

University of California, Berkeley

LiTER provides insights with targeted evaluation and aims to complement global evaluation metrics. LiTER should not be used with global/targeted metrics because lower LiTER can be achieved by more accurate idiom translations or hallucinations. 


LiTER

LiTER aims to help produce models that improve on LitTER without sacrificing general quality. LiTER is expected to be relevant when developing NMT models for creative content like subtitles, social media text. 


Learn Before

Related