Unsupervised NMT Components
Unsupervised NMT relies on two components to ensure learning efficiency and quality:
- Bilingual alignment
- Translation improvement


Unsupervised NMT Components for leveraging monolingual data for Low-Resource NMT

Unsupervised NMT deals with the zero-resource translation scenario where there are no parallel sentences whatsoever. It uses only monolingual data to get an initial alignment and improve the translation via iterative back translation.

Columbia University

The following approaches are used to leverage monolingual data for low-resource NMT:
- back translation and forward translation
- joint training on both translation directions
- unsupervised NMT
- pre-training
- exploiting comparable corpus
- enhancing with bi-lingual dictionary 

Methods for Leveraging Mono-Lingual Data for Low-Resource NMT

Wang, R., Tan, X., Luo, R., Qin, T., & Liu, T.-Y. (8 2021). A Survey on Low-Resource Neural Machine Translation. Στο Z.-H. Zhou (Επιμ.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 (σσ. 4636–4643). doi:10.24963/ijcai.2021/629


A Survey on Low-Resource Neural Machine Translation

In back translation, pseudo parallel sentence pairs are generated by translating the target-side monolingual sentences to the source language via a translation system in the reverse direction, while in forward translation, pseudo parallel sentence pairs are generated by translating the source-side monolingual sentences to the target language via a translation system in the same direction. Following that, the pseudo parallel data of either kind is used to train the NMT model. Back and forward translation have been shown to provide promising performance improvement on NMT systems. 


Back translation and forward translation for leveraging monolingual data for Low-Resource NMT

Considering that both the source and target sides may have monolingual data that has valuable information, one can leverage both at once via joint training on the two translation directions. This type of dual learning simultaneously improves the two models on both translation directions by aligning the original monolingual sentences $x$ and the sentences $x’$ translated forward and then backward ($x \to y’ \to x’$) by the two models. Dual learning can be further improved by introducing multi-agent for both translation directions. Iterative back translation simultaneously trains the NMT models on both translation directions and iteratively updates the back-translated corpus via the updated better NMT models. Bi-directional NMT trains both the translation directions in the same model with a tag indicating the direction at the beginning of source sentences, after which it leverages both source-side and target-side monolingual data by back and forward translation. Mirror-generative NMT jointly trains the translation models on both directions and the language models for both source and target languages with a shared latent variable. 


Joint training on both translation directions for leveraging monolingual data for Low-Resource NMT

Unsupervised NMT for leveraging monolingual data for Low-Resource NMT

Pre-training language models can be very helpful for NMT, especially in the low-resource setting. Language model pre-training initializes the NMT models with language understanding and generation capability using only monolingual data. It is possible either to train the encoder and/or decoder separately, or jointly. In terms of separate pre-training, you can separately initialize the encoder and the decoder with language models and then fine-tune with supervised parallel data. Its also possible to initialize the encoder and decoder with separate language models training by a combination of masked language modeling, where some tokens in the text are masked and the model learns to predict the masked tokens. Translation language modeling extends masked language models by concatenating parallel sentence pairs as the input sentences. A drawback of separately training encoder and decoder is that it cannot train the encoder-decoder-attention well, which is very important in NMT to connect the source and target representations for translation. Therefore, jointly pre-training the encoder, decoder, and attention is better for translation accuracy. This can be done by using masked sequence to sequence learning that randomly masks a fragment (several consecutive tokens) in the input sentence of the encoder, and predicts the masked fragment in the decoder. We can also add noises and randomly mask tokens in the input sentences in the encoder, and learn to reconstruct the original text in the decoder. T5, for example, randomly masks some tokens and replaces the consecutive tokens with a single sentinel token. 

Language model pre-training for leveraging monolingual data for Low-Resource NMT

Monolingual data of different languages that refers to the same entity (e.g. Wikipedia pages in different languages describing the same object) can be considered comparable corpuses, which are easier to obtain than parallel data.  Comparable corpora are strong supplements to parallel corpus, from which parallel sentences can be extracted based on language models or translation models. These corpuses contain implicit parallel information for NMT systems. The problem here is mining the parallel sentences from the comparable corpus. Cross-lingual sentence embeddings, extracting potential aligned target sentences given a source sentence and then making the target sentences better aligned with the source sentence by revising them via an editing mechanism, and self-supervised learning where finding semantically aligned sentences is considered an auxiliary task are ways to do this. In addition to mining parallel sentences, one can take advantage of the aligned topic distribution for weakly paired documents, which is suitable for documents related to the same event or entity but not implicitly aligned in sentences. 


Exploiting comparable corpus for leveraging monolingual data for Low-Resource NMT

Bilingual dictionary contains word-level parallel information, which is helpful on the alignment between two languages. The bilingual dictionary of a low-resource language paid can be collected either by human annotation or word embedding based alignment, which is much easier to obtain than the bilingual parallel sentences. Since the bilingual dictionary contains only word-level information, it is usually used with monolingual data to improve the translation. Pseudo-parallel sentences can be build by translating the source-side monolingual sentences to target language via SMT which is built based on the bilingual dictionary. Parallel data can be augmented by replacing some words in parallel sentences with rare words. Furthermore, bilingual dictionaries can also be used to perform word-by-word translation on monolingual data. Finally, the gap of embedding spaces between source and target languages can be bridged by anchoring points based on the dictionary. 

Learn Before

Related

Learn After