Parallel data is split into regular and idiom data using a pattern-matching tool built on Spacy’s (Hannibal and Montani, 2017) rule-based matching engine. The span of each idiom’s occurrence in a sentence is annotated. 


Idiom Data Training Splits 


The data (list of 225 English idioms) is fed into the pattern-matching tool, and translation pairs containing an idiom on the source side are extracted and annotated. Sentences with idioms occurring only once are discarded. Regular data is used for training, while the idiom data are evenly divided into idiom-train and idiom-test sets, i.e. half of each idiom’s sentence pairs go in the training set. 


Processing Idiom Data 


There are three idiom testing conditions.


Idiom Testing Conditions


The IWSLT dev-set for each language pair is used for development. For general translation evaluation, results are reported in the WMT newstest14 and IWSLT’17 test sets for en->fr and WMT newstest13 and IWSLT’17 test sets for en->es. The extracted idiom-test data per language pair is used for targeted idiom evaluation. A fast-align model trained on language-pairgenerates word alignments for APT-Eval, and beam search (beams size 5) and BLEU computed with SacreBLEU evaluates all models. 


Experiment Evaluation 


Sentence pairs 80+ words and length rations greater than 1.5 are filtered out. The authors use sentence piece(SPM, Kudo and Richardson, 2018) to tokenize remaining sentences. 


Idiom Data Preprocessing 


Pretraining on monolingual data improves generic idiom translation quality, but the don’t contain examples for translating idioms between languages. 


Pretraining Models on Monolingual Data 


The en->fr are consistent with en->es. Global evalution using BLEU on diverse test sets and two targeted evaluation methods are considered. For the upsampling split, the idiom-train data is upsampled 20x, and models started to overfit with 100x upsampling. 


Idiom Model Results 


Authors use annotations in the idiom data to encode idiom words within different contexts, measuring how it affects decoder distributions and translation output. (1) full context encodes the idiom phrase within the whole input sentence, (2) phrase-level context encodes idiom phrase in isolation, and (3) word-level context encodes each idiom word independently. 


Understanding how Models Translate Idioms


Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


University of California, Berkeley

Targeted evaluation isolates the idiom translation with word alignments or word edit distance, which measures idiom translation accuracy but doesn’t account for literal translation errors. Shao et al. (2018) proposed a method for estimating these errors’ frequency but it requires hand-crafted blocklists with words corresponding to literal translation errors. 


Targeted Evaluation for NMT of idiomatic expression 


Literal Translation Error Rate (LiTER) is a metric for a model’s literal translation errors. A literal translation error happens when a sentence is wrongly translated literally in the target language. Translation errors are identified by checking if a translation output contains any (automatically created compiled) blocklisted words. 


Automatic Targeted Evaluation 


Alignment-based Phrase Translation Evaluation (APT-Eval) extends Fadee et al. (2018) and subword-level metrics to measure idiom translation accuracy. It uses word alignments to find words in the hypothesis and reference sentences which align with the annotated idiom source span and compares the two. 

Alignment-based Phrase Translation Evaluation (APT-Eval) 


APT-Eval is a targeted evaluation metric which measures translation accuracy. It is unable to measure literal translation errors in inaccurate translations, but LiTER is able to. 


LITER vs APT-Eval


Prior research overlooks the fact that different idioms have significantly different frequencies. As a result, models overfitted on over-represented idioms are favored. To address this, targeted evaluation results are reported by macro-averaging over idioms. 


Learn Before

Related

Learn After