Focusing only on how models translate source-side idioms for targeted evaluation, the scores are macro-averaged to account for idiom frequency imbalance. Results are presented on LitTER metric and on APTEval, which provide different information. 


Targeted Evaluation Results 


All models produce fewer errors when trained on joint split compared to zero split. Pretraining and upsampling idiom-train data helps all models. Masking increases errors on the joint split, and decoder-side word replacements yield a similar behavior in terms of LitTer. Adding word replacements in the encoder reduces LitTER. 


Literal Translation Errors 


To estimate model accuracy, reference and hypothesis matches aligning with source idiom words are compared. Joint split improves accuracy, and pretraining outperforms random initialization. Upsampling doesn’t yield consistent improvements, and source-side word replacements reduce literal translation errors while degrading idiom translation accuracy. 


Idiom Translation Accuracy


mBART models are significantly better than random models, and there is no measurable difference between splits. Noisy finetuning methods do not improve results and encoder-side word replacements degrade overall performance. It is hypothesized that they induce hallucinations. 


General Purpose Results 


Global evaluation considers full sentences, which overshadows idiom translation impact. Models perform better when trained on joint split. 


Idiom-Test Results


The en->fr are consistent with en->es. Global evalution using BLEU on diverse test sets and two targeted evaluation methods are considered. For the upsampling split, the idiom-train data is upsampled 20x, and models started to overfit with 100x upsampling. 


University of California, Berkeley

Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

Parallel data is split into regular and idiom data using a pattern-matching tool built on Spacy’s (Hannibal and Montani, 2017) rule-based matching engine. The span of each idiom’s occurrence in a sentence is annotated. 


Idiom Data Training Splits 


The data (list of 225 English idioms) is fed into the pattern-matching tool, and translation pairs containing an idiom on the source side are extracted and annotated. Sentences with idioms occurring only once are discarded. Regular data is used for training, while the idiom data are evenly divided into idiom-train and idiom-test sets, i.e. half of each idiom’s sentence pairs go in the training set. 


Processing Idiom Data 


There are three idiom testing conditions.


Idiom Testing Conditions


The IWSLT dev-set for each language pair is used for development. For general translation evaluation, results are reported in the WMT newstest14 and IWSLT’17 test sets for en->fr and WMT newstest13 and IWSLT’17 test sets for en->es. The extracted idiom-test data per language pair is used for targeted idiom evaluation. A fast-align model trained on language-pairgenerates word alignments for APT-Eval, and beam search (beams size 5) and BLEU computed with SacreBLEU evaluates all models. 


Experiment Evaluation 


Sentence pairs 80+ words and length rations greater than 1.5 are filtered out. The authors use sentence piece(SPM, Kudo and Richardson, 2018) to tokenize remaining sentences. 


Idiom Data Preprocessing 


Pretraining on monolingual data improves generic idiom translation quality, but the don’t contain examples for translating idioms between languages. 


Pretraining Models on Monolingual Data 


Idiom Model Results 


Authors use annotations in the idiom data to encode idiom words within different contexts, measuring how it affects decoder distributions and translation output. (1) full context encodes the idiom phrase within the whole input sentence, (2) phrase-level context encodes idiom phrase in isolation, and (3) word-level context encodes each idiom word independently. 


Learn Before

Related

Learn After