Training data only includes regular parallel data. At testing, model performances on unseen idioms are measured. 


Zero Testing Condition


In neural machine translation experiments, the joint testing condition uses a training dataset containing both regular parallel data and a specific subset of idiom data (idiom-train data). During testing, the model's performance is measured on translating these same idioms, but within sentence contexts different from those seen in the training data.

Joint Testing Condition in Idiom Translation

In neural machine translation experiments, the upsampling testing condition is similar to the joint testing condition—which trains the model on a dataset containing both regular parallel data and a specific subset of idiom data (idiom-train data)—but the idiom-train data is upsampled $$N$$ times. This condition evaluates whether upsampling targeted training data is necessary or effective to achieve better idiom translation quality.

Upsampling Testing Condition in Idiom Translation

There are three idiom testing conditions.


University of California, Berkeley

Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

The data (list of 225 English idioms) is fed into the pattern-matching tool, and translation pairs containing an idiom on the source side are extracted and annotated. Sentences with idioms occurring only once are discarded. Regular data is used for training, while the idiom data are evenly divided into idiom-train and idiom-test sets, i.e. half of each idiom’s sentence pairs go in the training set. 


Processing Idiom Data 


Idiom Testing Conditions


The IWSLT dev-set for each language pair is used for development. For general translation evaluation, results are reported in the WMT newstest14 and IWSLT’17 test sets for en->fr and WMT newstest13 and IWSLT’17 test sets for en->es. The extracted idiom-test data per language pair is used for targeted idiom evaluation. A fast-align model trained on language-pairgenerates word alignments for APT-Eval, and beam search (beams size 5) and BLEU computed with SacreBLEU evaluates all models. 


Experiment Evaluation 


Pretraining on monolingual data improves generic idiom translation quality, but the don’t contain examples for translating idioms between languages. 


Pretraining Models on Monolingual Data 


The en->fr are consistent with en->es. Global evalution using BLEU on diverse test sets and two targeted evaluation methods are considered. For the upsampling split, the idiom-train data is upsampled 20x, and models started to overfit with 100x upsampling. 


Idiom Model Results 


For idiom translation experiments, sentence pairs with more than 80 words and length ratios greater than 1.5 are filtered out. The remaining sentences are tokenized using SentencePiece (SPM) (Kudo & Richardson, 2018).

Idiom Data Preprocessing

To analyze how neural machine translation (NMT) models represent and translate idioms, researchers vary the amount of surrounding context available to the encoder using three context-masking conditions:

1. **Full Context**: The idiom phrase is encoded normally within the entire input sentence, allowing the model to use surrounding syntactic and semantic clues.
2. **Phrase-Level Context**: The idiom phrase is encoded in isolation (with the surrounding sentence context masked or removed), forcing the model to represent the idiom solely based on its constituent words.
3. **Word-Level Context**: Each word of the idiom is encoded independently, breaking phrase-level cohesion and forcing a compositional, word-by-word representation.

By feeding these encoder representations to the decoder, researchers can measure how varying context affects decoder distributions and translation output.

Context Levels for Probing NMT Idiom Translation

Parallel data is split into regular and idiom subsets using a pattern-matching tool built on Spacy's (Hannibal and Montani, 2017) rule-based matching engine. The span of each idiom's occurrence in a sentence is annotated.

Learn Before

Related

Learn After