Reducing idiom context results in worse translation scores, with full-context pretraining yielding the best results. While reducing idiom context barely affects random models, it is catastrophic for pretrained models. This implies that pretrained models are more global, containing information related to surrounding idiom context, whereas random models are myopic and rely on information related only to the idiom tokens themselves.

Reducing Idiom Context

This probe measures how translation quality changes when only the idiom words' encoder representations are altered. Each full sentence is first encoded normally, so non-idiom words retain full-sentence context, but the representations of the idiom words are replaced with representations obtained under a narrower context (phrase-level or word-level, as defined in the parent probing setup). The resulting hybrid encoding is decoded, and the translation output is compared against the reference translations to measure the resulting change in translation performance.

University of California, Berkeley

Google

To analyze how neural machine translation (NMT) models represent and translate idioms, researchers vary the amount of surrounding context available to the encoder using three context-masking conditions:

1. **Full Context**: The idiom phrase is encoded normally within the entire input sentence, allowing the model to use surrounding syntactic and semantic clues.
2. **Phrase-Level Context**: The idiom phrase is encoded in isolation (with the surrounding sentence context masked or removed), forcing the model to represent the idiom solely based on its constituent words.
3. **Word-Level Context**: Each word of the idiom is encoded independently, breaking phrase-level cohesion and forcing a compositional, word-by-word representation.

By feeding these encoder representations to the decoder, researchers can measure how varying context affects decoder distributions and translation output.

Context Levels for Probing NMT Idiom Translation

Using more context and pretraining improves (lowers) perplexity. Training on joint split consistently improves random models, and upsampling idiom-train data yields bad results due to overfitting. 


Increasing Idiom Context 


Token-level uncertainty varies in idiom vs non-idiom word translation. The authors first translate each sentence pair with teacher-forcing, measures the entropy of the decoder’s distributions for each target token. Lastly, they use word alignments to separately average the entropy values of target words assigned to idiom and non-idiom source words. 


Decoder Uncertainty


Variation in Translation Performance

This probe measures how varying the encoder's idiom context (full-context, phrase-level, or word-level, as defined in the parent probing setup) affects the likelihood the model assigns to the correct reference translation. For each context condition, the reference translation is scored by its perplexity under the model, conditioned on the corresponding encoder output sequence; lower perplexity indicates that the encoder representation makes the reference translation more likely.

Learn Before

Related

Learn After