In this probe, the authors translate encoder representation and evaluate the sample against the reference translations. Each full input sentence is encoded, and those of the idiom words are replaced with those obtained with narrower contexts. 


Variation in Translation Performance


In this probe, authors vary encoder idiom representation to measure how it affects the likelihood of reference translation. Each reference translation is scored by perplexity under the model given each encoder output sequence. 


Variation in Translation Likelihood 


Using more context and pretraining improves (lowers) perplexity. Training on joint split consistently improves random models, and upsampling idiom-train data yields bad results due to overfitting. 


Increasing Idiom Context 


Token-level uncertainty varies in idiom vs non-idiom word translation. The authors first translate each sentence pair with teacher-forcing, measures the entropy of the decoder’s distributions for each target token. Lastly, they use word alignments to separately average the entropy values of target words assigned to idiom and non-idiom source words. 


Decoder Uncertainty


Authors use annotations in the idiom data to encode idiom words within different contexts, measuring how it affects decoder distributions and translation output. (1) full context encodes the idiom phrase within the whole input sentence, (2) phrase-level context encodes idiom phrase in isolation, and (3) word-level context encodes each idiom word independently. 


University of California, Berkeley

Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

Parallel data is split into regular and idiom data using a pattern-matching tool built on Spacy’s (Hannibal and Montani, 2017) rule-based matching engine. The span of each idiom’s occurrence in a sentence is annotated. 


Idiom Data Training Splits 


The data (list of 225 English idioms) is fed into the pattern-matching tool, and translation pairs containing an idiom on the source side are extracted and annotated. Sentences with idioms occurring only once are discarded. Regular data is used for training, while the idiom data are evenly divided into idiom-train and idiom-test sets, i.e. half of each idiom’s sentence pairs go in the training set. 


Processing Idiom Data 


There are three idiom testing conditions.


Idiom Testing Conditions


The IWSLT dev-set for each language pair is used for development. For general translation evaluation, results are reported in the WMT newstest14 and IWSLT’17 test sets for en->fr and WMT newstest13 and IWSLT’17 test sets for en->es. The extracted idiom-test data per language pair is used for targeted idiom evaluation. A fast-align model trained on language-pairgenerates word alignments for APT-Eval, and beam search (beams size 5) and BLEU computed with SacreBLEU evaluates all models. 


Experiment Evaluation 


Sentence pairs 80+ words and length rations greater than 1.5 are filtered out. The authors use sentence piece(SPM, Kudo and Richardson, 2018) to tokenize remaining sentences. 


Idiom Data Preprocessing 


Pretraining on monolingual data improves generic idiom translation quality, but the don’t contain examples for translating idioms between languages. 


Pretraining Models on Monolingual Data 


The en->fr are consistent with en->es. Global evalution using BLEU on diverse test sets and two targeted evaluation methods are considered. For the upsampling split, the idiom-train data is upsampled 20x, and models started to overfit with 100x upsampling. 


Learn Before

Related

Learn After