Models have significantly higher uncertainty when translating idiom words vs regular words, meaning idioms are harder to translate. When translating regular words, random models are unaffected by changing idiom representations. However, reducing idiom context increases the pertained model’s uncertainty. For idiom translation, upswing the idiom-train data benefits the pertained but not the random model. 


Idiom vs Regular Translation 


To compare decoder uncertainty for idiom and non-idiom translation, each target sentence is decoded under teacher forcing, and the entropy of the decoder distribution is computed at every target token. Word alignments then divide the target tokens according to whether they align with idiom or non-idiom source words, and mean entropy is calculated separately for the two groups.

University of California, Berkeley

Google

To analyze how neural machine translation (NMT) models represent and translate idioms, researchers vary the amount of surrounding context available to the encoder using three context-masking conditions:

1. **Full Context**: The idiom phrase is encoded normally within the entire input sentence, allowing the model to use surrounding syntactic and semantic clues.
2. **Phrase-Level Context**: The idiom phrase is encoded in isolation (with the surrounding sentence context masked or removed), forcing the model to represent the idiom solely based on its constituent words.
3. **Word-Level Context**: Each word of the idiom is encoded independently, breaking phrase-level cohesion and forcing a compositional, word-by-word representation.

By feeding these encoder representations to the decoder, researchers can measure how varying context affects decoder distributions and translation output.

Context Levels for Probing NMT Idiom Translation

This probe measures how translation quality changes when only the idiom words' encoder representations are altered. Each full sentence is first encoded normally, so non-idiom words retain full-sentence context, but the representations of the idiom words are replaced with representations obtained under a narrower context (phrase-level or word-level, as defined in the parent probing setup). The resulting hybrid encoding is decoded, and the translation output is compared against the reference translations to measure the resulting change in translation performance.

Variation in Translation Performance

This probe measures how varying the encoder's idiom context (full-context, phrase-level, or word-level, as defined in the parent probing setup) affects the likelihood the model assigns to the correct reference translation. For each context condition, the reference translation is scored by its perplexity under the model, conditioned on the corresponding encoder output sequence; lower perplexity indicates that the encoder representation makes the reference translation more likely.

Learn Before

Related

Learn After