Multilingual language models (LMs) are typically trained in two phases: a “pre-training” phase, in which unlabeled data from one or multiple languages may be used, and a “fine-tuning” phase, in which task-specific labeled data is used to build a system capable of solving a target task.

Two-Phase Training of Multilingual Language Models

 multilingual BERT have shown promise in enabling the coverage of lowresource languages without any labeled data by using the zero-shot framework

University of Michigan - Ann Arbor

language technologies are not yet capable of processing C-S as seamlessly as monolingual data. 

We identify three main limitations of the current state of computational processing of C-S: data, evaluation and user-facing applications.

Computational Approaches to C-S

Massive multilingual Language Models (LMs)

While multilingual BERT outperforms older task-specific models on C-S tasks, its performance on C-S data is significantly worse than on the same tasks in a monolingual setting. This inefficiency occurs because these models are either not exposed to C-S data at all during training, or they miss out on several language pairs.

Inefficiency of Multilingual BERT on C-S Tasks

Since C-S languages tend to be low-resourced, building deep learning-based models is challenging due to the lack of large C-S datasets.

Learn Before

Related

Learn After